Guest Lecture by Bill Inmon

Thanks to my good friend professor Bill Slater, I was able to attend a guest lecture by Bill Inmon last night on the topic of data lakes.

20160629_201048

Who is Bill Inmon?
https://en.wikipedia.org/wiki/Bill_Inmon
William H. Inmon (born 1945) is an American computer scientist, recognized by many as the father of the data warehouse.[1][2] Bill Inmon wrote the first book, held the first conference (with Arnie Barnett), wrote the first column in a magazine and was the first to offer classes in data warehousing. Inmon created the accepted definition of what a data warehouse is – a subject oriented, nonvolatile, integrated, time variant collection of data in support of management’s decisions. Compared with the approach of the other pioneering architect of data warehousing, Ralph Kimball, Inmon’s approach is often characterized as a top-down approach.

Pandoc – Argument of \Paragraph has an extra }

Writing a textbook for a class — as I am doing — is hard enough, I am using Pandoc for my choice of markdown to produce PDF and ePub.

There are differences and trade offs trying to take a common markdown code base and publish it to different formats. Currently I don’t have a feature parity between both formats they both have extra items the other one doesn’t.

Recently I reinstalled Pandoc with MikTeX for Windows and BasicTeX for Mac, and deb package texlive for Ubuntu 16.04. These packages are needed to turn Pandoc markdown into PDFs via a LateX library.

Installing the latest version of Pandoc 1.17.1 and running my build script to produce a PDF all of a sudden generated this error…

pandoc-error

Oh no… this error is not very helpful as I have 13 chapters worth of material, what could be causing this error? It could be anywhere! I did what any good detective did, I removed each chapter 1 by 1 and rebuilt the PDF trying to narrow down where the error is by process of elimination.

I went all the way down – stripping all the chapters out but one. That was the title page. The file titlesec.tex is a LaTeX style formatting command to generate a title page and also make sure that each new chapter starts on its own page. This worked for the entire last year using this command line option:

Here is the full code that throws the error:

pandoc --toc -V geometry:margin=1in --number-sections --include-in-header ./title/titlesec.tex -s -o ./output/pdf/Understanding-the-Technology-and-Philosophy-of-Linux-Part-I-$STAMP.pdf ./Chapter-01/chapter-01.md ./Chapter-02/chapter-02.md ./Chapter-03/chapter-03.md ./Chapter-04/chapter-04.md ./Chapter-05/chapter-05.md ./Chapter-06/chapter-06.md ./Chapter-07/chapter-07.md ./Chapter-08/chapter-08.md ./Chapter-09/chapter-09.md ./Chapter-10/chapter-10.md ./Chapter-11/chapter-11.md ./Chapter-12/chapter-12.md ./Chapter-13/chapter-13.md ./Chapter-14/chapter-14.md ./Chapter-15/chapter-15.md ./Appendix-A/Appendix-A.md ./Appendix-B/Appendix-B.md ./Appendix-C/Appendix-C.md

Strange. The quick fix for this error was to remove the offending line of code:

--include-in-header ./title/titlesec.tex

Replacing it with this line:
-V documentclass=report

Which then generates the expected PDF — compare the difference below:

pandoc-broke

pandoc-fix

pandoc-success

Video of First Successful Prototype of the Autonomous Movement Framework (Drones and UAV)

This was done in March of 2015 — I forgot to post it so I am posting it now.

Fist successful flight of the Autonomous Service Framework

The voice you hear is then Undergraduate student Mark Milhouse — completing this demo out in the February snow of 2015, near the IIT Keating Sports Center.

This is the first outtake video of our unit getting confused and dropping its payload mid-air

Finally this last video is some funny out-takes of the units crashing, all set to music.

This was all the footage from our initial round of research.

Eucalyptus 4.0.2 supports using AWS Java SDK 1.11.4 for Walrus but really 1.9.4

225px-Eucalyptus-Logo

Here at Illinois Tech we have just completed a successful semester of using HP Eucalyptus 4.0.2 Cloud Platform for all of our capstone and research projects. It has been a blessing as it has taken hardware requests and budgeting problems out of the IT groups hands and placed it into the students hands. They were able to rapidly deploy their services, load-balancers and any resources needed on their own.

Still in the process of upgrading to Eucalyptus 4.0.2 but it still works with S3cmd and the AWS Java API (as they intended it to).

This article is about Eucalyptus 4.0.2 — I still haven’t done the upgrade, 4.2.2 adds a few new features (looking at you UFS that I need to build in a test environment. I also need to upgrade my old HP DL 360s which have been work horses but the memory is only 4GB so that is asking for trouble.

This article is about getting the AWS Java SDK working with Eucalyptus 4.0.2. I was able to make this work… with conditions. The first issue is to review current information on hand. A quick search of the old Eucalyptus GitHub wiki gives some clues but only talks about ASW Java SDK version 1.3.x and Eucalyptus version 3.3.x. Of use for a few clues.

The next place to search is the Eucalyptus Users Google Group, which I recommend to visit if you have a Eucalyptus issue, very friendly people who love the product. I executed this search string: https://groups.google.com/a/eucalyptus.com/forum/#!searchin/euca-users/AWS$20Java$20SDK and a few interesting results came up. (Note one of the questions is my own…)

aws-java-compat-list

Credit goes to HP Eucalyptus employee, Steve Jones and user CoderSparks, who noticed that Walrus doesn’t support signature v4 in the AWS Java SDK — you need to default back to the version 2 signature — just like in the previous S3cmd article.

I wanted to see for myself what the issue was so I tried to use the latest AWS Java SDK v1.11.4. Surprisingly everything worked, but one important feature. Let’s find out why.

Development Environment

My setup was:

  • Windows 10 machine
  • Netbeans 8.1
  • AWS Java SDK GitHub repo (contains the S3Sample file and Maven project build files)
  • EDIT 07/13/16 – Clone this project: https://github.com/awslabs/aws-java-sample.git
  • –EDIT 06/16/16–
    I opened the sample/test code provided by Amazon, located in the https://github.com/aws/aws-sdk-java/blob/master/src/samples/AmazonS3/S3Sample.java file.

    First start by cloning the AWS Java SDK sample projects from AWSlab:

    git clone https://github.com/awslabs/aws-java-sample.git

    Second you can then open the project directly in NetBeans and it will import the entire build package (S3Sample.java) plus the Maven file (pom.xml) required to download all the .jar dependencies. Just hit the green RUN triangle and Maven will retrieve all of the JARs needed, even the AWS Java SDK.

    Issues Encountered
    I encountered one small issue using AWS Java API 1.11.4 — this issue is referenced here by member CoderSparks. – though the questioner had a different error — I think the concept was the same:

    What I Did Next
    Let me show you the initial code, my additions, and then explain it below. Modify the first few lines of code in the sample to reflect below.

    // My additions to the S3Sample.java code 
    ClientConfiguration opts = new ClientConfiguration();
    opts.setSignerOverride("S3SignerType");  // NOT "AWS3SignerType"  -- mentioned by Steve Jones set signatures back to v2.
            
    AmazonS3Client s3 = new AmazonS3Client(opts);
    // Region usWest2 = Region.getRegion(Regions.US_WEST_2);  
    // s3.setRegion(usWest2);
    s3.setEndpoint("http://objectstorage.yourdomain.com:8773/services/objectstorage/");
    s3.setS3ClientOptions(new S3ClientOptions().withPathStyleAccess( true ) );
    

    Also don’t forget to create a credential file on the local system–don’t put creds in your actual code–too risky. In your home directory (Linux or Windows), create a directory named .aws with a file named credentials inside that file place the syntax below. These keys can be AWS account keys or HP Eucalyptus Account keys. The keys will come in your credentials zip file provided by your Eucalyptus Administrator, extract the zip file and the credentials listed below will be contained in a file named eucarc. In this story they are your account keys from our HP Eucalyptus system.

    [default]
    aws_access_key_id = YOURACCESSKEYHERE
    aws_secret_access_key = YOURSECRETKEYHERE

    This is how the S3 connection object looks in the S3Sample.java provided in the Github link for the AWS JAVA SDK project.

    AmazonS3 s3 = new AmazonS3Client();
    Region usWest2 = Region.getRegion(Regions.US_WEST_2);
    s3.setRegion(usWest2);
    

    Eucalyptus has different regions so to speak. So you need to comment out a few AWS specific lines and add an override so the code knows how to find your Walrus/OSG (Object Storage Gateway) URI.

    AmazonS3 s3 = new AmazonS3Client();
    //Region usWest2 = Region.getRegion(Regions.US_WEST_2);
    //s3.setRegion(usWest2);
    s3.setEndpoint("http://objectstorage.yourdomain.com:8773/services/objectstorage");
    s3.setS3ClientOptions(new S3ClientOptions().withPathStyleAccess( true ) );
    

    With this set, the code will compile and run and connect to AWS Java SDK 1.11.4, and all of the JAR dependencies will be fetched by Apache Maven for you, easy setup no need to download any additional JARs or classpath.

    From Whence Came That Error?
    I am glad you asked. I though it might be the same issue as the in the S3cmd article… Let’s look at the sample code and go item by item until we get to the error and squeeze its meaning out.

    • Proper Connection being made to Eucalyptus OSG (Walrus) Green-Tick
    • Proper code that creates a new bucket Green-Tick
    • Proper code that creates uploads an new object to the previously created bucket Green-Tick
    • Proper code that downloads that previously uploaded object th

    Error as follows from the exception handling in Java:

    Downloading an object
    May 31, 2016 12:37:06 AM com.amazonaws.services.s3.AmazonS3Client getSignerByURI
    WARNING: Failed to parse the endpoint http://objectstorage.sat.iit.edu:8773/services/objectstorage, and skip re-signing the signer region
    java.lang.IllegalArgumentException: Invalid S3 URI: hostname does not appear to be a valid S3 endpoint: http://objectstorage.sat.iit.edu:8773/services/objectstorage

    sampleS3-v3-error

    sampleS3-v3-error-P2

    What could be causing this error? The connection is proper and 3/4ths of the code works

    It is the signature problem mentioned above but also something else. The AWS Java SDK is too new and the way it handles downloading an object is not compatible. This means I had to roll versions back. If you roll back to AWS Java SDK 1.9.4 then you get 4 green check marks.

    TL DR; Solution

    • Add the setSignature() method detailed above
    • Change the POM.xml file value to pull AWS Java SDK 1.9.4 (no higher)
    • Walrus problems are solved

    pom-xml-1-9-4

    Take care
    Jeremy!

Using S3cmd with Walrus and HP Eucalyptus 4.0.2 works but with a twist…

Dear Team,

I would like to pass on that using S3cmd works fine with Eucalyptus 4.0.2 (will be installing and testing 4.2.2 soon) with a twist… Assuming you are using Linux or the Bash on Windows 10 (Linux subsystem) for this exercise…

S3cmd is a commandline tool built in Python for accessing Object Storage via the commandline. This is useful for AWS S3, but also for HP Eucalyptus Walrus (Eucalyptus S3 Object storage–AWS compatible).

s3cmd-permission-denied

My command was as follows: ./s3cmd -v -c ./s3cfg la s3://vagrant-builds

A bucket I created with my account credentials should have list and read permission. Even switching to the ADMIN credentials the same error appears.

Then I noticed a little tidbit in the S3cmd –help file that said this:

signature-v2

Look at the entry that says:
--signature-v2 Use AWS Signature version 2 instead of newer signature methods. Helpful for S3-like systems that don't have AWS Signature v4 yet.

velma_jinkies_by_tinent

Thats right Velma, jinkees, a clue.

AWS changes their API often and certain aspects of their code changes that invalidates using old APIs–which I guess makes sense from their upkeep point of view. BUT it makes it hard for honest Eucalyptus developers that are trying to peg their product to AWS Java SDK, so it seems they are a few signature versions back.

How to generate s3cfg

These are detailed links to S3cmd and its setup.

Copy the text below and place it in a file named .s3cfg in your home directory.

[default]
access_key = YOURKEYHERE
secret_key = YOURSECRETKEYHERE
host_base = xxx.xxx.xxx.xxx:8773
host_bucket = xxx.xxx.xxx.xxx:8773
service_path = /services/objectstorage
bucket_location = US
default_mime_type = binary/octet-stream
delete_removed = False
dry_run = False
enable_multipart = True
encoding = UTF-8
encrypt = False
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase = password
guess_mime_type = True
human_readable_sizes = False
invalidate_on_cf = False
list_md5 = False
log_target_prefix =
mime_type =
multipart_chunk_size_mb = 15
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
recursive = False
recv_chunk = 4096
reduced_redundancy = False
send_chunk = 4096
skip_existing = False
socket_timeout = 300
urlencoding_mode = normal
use_https = False
verbosity = WARNING

Adding this flag to the S3cmd solves the long listing problem using my own credentials.
./s3cmd -v --signature-v2 -c ./s3cfg la s3://vagrant-builds

Now the results are what I was looking for!

s3cmd-works

SAT summer break IT projects – volunteers needed @ Rice Campus

Dear Team,

How are you? Just to show some snapshots of what we are working on to get hands on experience setting up some networking and installation projects. No experience is needed–will train–only a willing mind.

take care
Jeremy Hajek

Euca2ools 3.1.x works on Bash for Windows 10 14323

Bash for Windows or the Linux Subsystem for Windows

Since it is basically Ubuntu 14.04 – euca2ools should install properly. I set out to see what would happen as it would be comforting to be able to use the Linux subsystem for euca2ools work.

bash-git-clone

Following the normal process — everything worked! But one thing…
bash-euca-version

bash-on-win-install

The thing to keep in mind is that Ubuntu 14.04 has its own version of the python-requests library. You need to remove this version:

sudo apt-get remove python-requests

bash-on-win-install

bash-euca2ools-building

bash-euca-describe-instances

May 5th WGN TV interview

In case you missed it last night, Jeremy’s Hajek’s interview appeared during the 9pm newscast on WGN TV last Great job Jeremy!

Link to presentation:
The evolution of the Smartphone

Just a side note, since this is the critical news rating time period for television stations (future advertising rates are set based on ratings for this month) news stories are carefully selected to generate maximum ratings. WGN News is the number one newscast in this time period. Appearing in a news story during this broadcast is significant.

DevOps Meetup w/Bryan Cantrill and Ray Krueger

Dear Friends,

Went to an amazing DevOps meetup here in Chicago and heard Bryan Cantrill (CTO of Joyent) speak. He is as advertised. Also heard from Ray Krueger, the CTO of Hilton Hotels talk about their own IT and infrastructure patterns.

They posted a video of the presentation here:

Go to about 1:40 (1 hour 40 minutes in) to hear me ask a question about debugging and tracings place in education.

I highly recommend joining and attending the DevOps Chicago meetup: (free and they usually feed you)

DevOps Chicago

Chicago, IL
1,560 DevOps professionals

This is a group of professionals (and hackers), managers, engineers (both dev and ops) and entrepreneurs coming together to learn about and share information on DevOps practic…

Next Meetup

Nicole Forsgren – My First Year at Chef: Measuring All the T…

Thursday, May 26, 2016, 5:30 PM
99 Attending

Check out this Meetup Group →