MemTransfer and Exploring the Potential of Gigabit

The Background

It’s summer and I have free time, so it means lots of projects exploring thoughts I have had for the past few months and haven’t had the time to expand upon. This evenings lucky idea is really just an application of a few concepts I have learned this past semester in my computer hardware systems course (CIS450) at Kansas State University. I have always been fascinated with optimizations of anything and everything computer science related. Frustrated by the bottleneck big ole data drives cause, I wanted to explore what are the true limits of a gigabit connection.

The Hardware Setup

The physical setup for this experiment was a client (my personal 2013 MacBook Air, 1.7 Ghz i7, 8GB ram, 128 GB SSD) connected to my house’s gigabit network infrastructure via a USB3-to-Gigabit adapter, and my families fileserver, a Dell Core2 Quad, 6 GB ram, and loads of disks, also connected via gigabit. Props to the USB3-to-Gigabit adapter for handling the test so well, it could potentially be a bottleneck in the only achieving 600-650 Mb/s, as opposed to the theoretical 1,000 Mb/s (though overhead has to go somewhere in there!).

The Software Setup

The software for this system was a simple client and server written by me in Java. It is available on Github. The client takes a file as input, loads it completely into memory. Files over 4-5GB aren’t really feasible in this method, at least with these machines. It then sends it on its merry way to the server, who was preallocated a place in memory of the correct size. Once it has finished transferring, it marks the transfer as done, and writes it to disk. The key measurement is just the transfer time and not any of the reading from or saving to disk.

The Results

This first image simply shows the spike in receiving speeds on the server, all the way up to 608 Mbps. I have never seen speeds this fast transferring to one of the SATA drives. An interesting note is the sending ~10-15 Mb/s. It is hypothesized as TCP ACKing later on in the post.

As the file transfer happens all in memory, you can see the noticeable bump that comes from allocating the 1.4 GB test file, and the prompt return back down.

This image shows the output of the server client, showing that it received the file, then once all received was done it save the file. About a 20 MB/s difference between the two.

This last image shows the client, sending out a file, then finishing.

What’s Next

While I feel pretty successful about this test, especially since, blogpost included it has taken around 2 hours, I do feel like there are some interesting ways it could be improved. Utilizing multiple NIC’s would be interesting, to attempt to reach 2 Gigabits (or the more practical 1.2 Gigabits (600 Mb/s x 2)). Also using UDP as opposed to TCP can yield faster speeds in large file transfer, it would be interesting to see how that would apply here. You can note that in the network diagnostic image, while it is receiving a large 600 Mb/s, it is also sending back a not that small 15 Mb/s. That’s a lot of ACKs. If we weren’t waiting on so many (as in UDP), what would that mean for results? Interesting questions for another time.


Data Structures, and why you need to actually learn them

This year, I am taking CIS 300 at Kansas State University. At first, I expected this class to be boring as it was a class on “data structures.” How much more boring could it get? Oooh, studying the differences in implementations between LinkedList and ArrayLists. The first day our professor proved to us just how wrong we were. He showed us two snippets of code:

String results = "";
for(char c: String.toCharArray()) {
results += (char)(c+3);


StringBuilder results = new StringBuilder();
for(char c: String.toCharArray()) {

He then proceeded to convert the entire collection of works by Shakespeare. Method 1 was estimated to take over 21 days by the computer. Method 2 finished in seconds. That caught my attention.
There are several cool data structures and search techniques that I have found awesome throughout this semester.

Read more

You Know You’re An AP Student When…

These are just a collection of my favorite “You know you’re an AP Student when,” with some contributions from myself. Just an idea of my schedule for my junior year, AP Calc AB, AP Language and Composition, AP Physics B, AP US History, and CIS 300 (Fall) and CIS 301 (Spring) at K-State. Plus a (programming) job, debate (policy, every weekend), club swimming, and high school swimming.

  • You often look up #apush, #aplang, #apphysics, #apcalc on Twitter just to see your fellow sufferers.
  • You don’t think of it as “cheating”, you think of it as “cooperative learning.”
  • You have multiple breakdowns throughout the year from school-related stress.
  • On top of your full schedule, you’re involved in several extra-curricular activities and you have a job.
  • Every extra-curricular activity you sign up for is framed with the words “It’ll look good for college.”
  • You sign up for a class because it’s weighted, not because the subject interests you.
  • You consider any non AP class to be a blow-off class.
  • You never meet anyone new in your classes because its been the same group of kids since ninth grade.
  • You and your closest friends are all in the top 10% of your class.
  • The probablity of your homework getting done depends more on your mood than the fact that it needs to be done.
  • You consider anything below a A to be a bad grade.
  • Read more

Verifying an Google Apps Alias Domain

So I’ve been working on a project for a while with Google App Engine and it has gotten close to being finished, so I decided to buy a domain name and put the project on it. Sounds simple and a 5 minute ordeal right? Wrong. Dead wrong.

So I need to verify it, the main and recommended way is to use TXT record supported by most domain name registrars. Well mine wasn’t supported. Shouldn’t be a problem, all I need to do is upload an HTML file to a server behind the domain name to verify. That is quite dificult it turns out.

Next I fire up a VM, put a web server on it, put the verification file on it, and open it to the world and put it in the DNS settings for the domain name. Cox Communications doesn’t allow port 80; FAIL.

Try two, I’ll do a redirect from my domain name to a file on projects. FAIL, Google Apps doesn’t allow redirects when looking for the verification file.

3rd times the charm, right? I’ll CNAME my App Engine and put my project there. FAIL, the App Engine just redirects me to

4th time I try to CNAME my own website. This has a similar result, getting redirected back to, the owner of the servers website.

5th time, I contacted my friend Brent and asked him to work some config magic so that I could CNAME my project server without redirecting. Finally I can validate!

Yet I am still waiting for that CNAME change to propogate to the DNS servers around the world… 1and1 is famous for their slowness. It shows up right on my computer I am just waiting for it to show up right at Google.

Never thought something so simple would take so long.

Happy Birthday Facebook App

Whenever I finish a new project I always post about it. Here I go.

The Happy Birthday Facebook App was designed for lazy people, me in mind, that want to post on peoples walls at midnight for their birthday so it can be special and all, but for one reason or another they can’t. This app basically lets you choose friends to wish happy birthday to at midnight, set a custom message, and adds it to the database to wait for that special night when it strikes 12. Then it will use that access token you gave it long ago and write on their birthday giving them your special message.

Like a large majority of my projects, it is open sourced.  You can find it at GitHub and do whatever you want with the source. If you are looking for a simple Java Facebook app example or a simple Java App Engine app example, then it’s for you.

Try it out on Facebook.

Eclipse Plugin Update Logger

If you have read my site for a while, or know what projects I am involved in, you know that for several summers I always do a little update on an Eclipse Plugin download logger. It started two years ago, then it just literally increased an integer in a text file for every download. You could only see how many times a thing had been downloaded. Last year I upgraded it to two separate versions. Initially I wrote a logger that logged to an XML file. It kept a date, IP, and what was downloaded. This was infinitely better however when used it production it had a flaw where there were multiple file handles editing the file at the same time and it wiped the file. So later that summer I changed about 1/3 of the code to instead of logging to XML, to log to a MySQL database. It included a great backend that would provide Geo-location based on the IP address,  other WHOIS stats on the IP, and various other information. It’s backend was a copy of my old CMS for this website so it even had a decent looking backend. It allowed you to download your log in a CSV format for whatever purposes you wanted. It was great, just not so polished and debugged.

This year I moved the project to GitHub and you can find it here. This years version features an install script that dramatically makes it simpler to setup. Once you clone the git repository just running the install.php will setup everything from the MySQL DB, to error reporting, to your Google Maps API (for the map on the geo-location). All that’s needed to install this correctly is giving the web server the ability to write/edit files.

If you have a plugin for eclipse that you release check it out. Worst case scenario is it breaks, and if that happens it will turn itself off, stop logging, and allow the downloads to continue as if it was never there. Check it out.

One Locked Down Laptop

I was recently watching a YouTube video, about how a guys computer got stolen (and how he recovered it using SSH, VNC, and hacker skills!) and all the lack of security he put on it and all the potential damages it could have caused. It made me think, on my new laptop, I had absolutely no security, not even an account password (normally I do have one.) With me going more and more places with my laptop, starting college classes again Fall 2011, and going to college full time as a freshmen in 2013, laptop theft is a strong probability. That being so, I spent about a day just locking down my laptop and seeing how much data I could protect, some of those things also protect my desktop now too.

Read more

The Facebook Graph and OAuth 2.0

During my recent Facebook app development using Java, I had to have some minor integration with Facebook, and I am here to share what I have learned and the cool parts about Facebook’s “Graph.”

Introduction to Working with Facebook

When writting my application, it happened to be a canvas app, or it was accessed via with a iframe to my acctual app. When I would get a new user I would immediatley get a new cookie that was named fbs_APPID within this cookie, most importantly, there was a access token. Now to know any information about a user, one must have one thing, an OAuth access token.

Getting an OAuth access token

Most times you will have a token in the cookies, however on some random times that I never figured out the pattern to, you just don’t have that information there. And as for most apps, you need to know some information about a user, and possibly even post or something of the like with the users permission. However, for protection, Facebook just doesn’t allow any application developer access to their entire database. So to allow only certain access to certain applications, applications are given OAuth tokens, these tokens are user specificic and generally session specific. With these tokens, you could use the later discussed Facebook Graph to get just about any needed data.
The process of getting an OAuth token if it is not in the “fbs_” cookie can be quite strenuous. First you need to redirect the user to the Graph server with certain application-specific parameters. Next, one of the parameters happened to be a call back url, now that url is called and Facebook gives you a code. Using this code and your application secret you can finally get a new OAuth access token. Basically what all these redirects does is allow Facebook to check that this user is acctually logged in and that this user is allowing the application access to this information.

Read more

Java – The Power Language (for Web or Console)

Recently, I wrote my first Facebook application. It was a small project, and it’s main purpose was to replace the common note entitled something like “Xth grade Schedule!!!” and it was designed to be more efficient for the end user. After inputting your schedule, you would go through and see who is in your classes (that the system already knows). There were other cool features like using some Facebook Social Plugins like the “Like” button and Facebook comments.

Now for this application, as I wrote it in a matter of days and released it the day of schedule releases, I had to go with a language I knew very well, and I had to have a reliable, dependable source to host it at. I automatically though of the Google App Engine as it is pretty sweet to play around with and now would be a great time to test it in a live environment. My previous application I had wrote on Google’s free, scalable engine was written in Python. It utilized the amazing Django framework, but I felt like I didn’t really understand the Python language as well as some others. The only other option for selection on the App Engine was Java. This was perfect as I had just completed a summer class on Java and it was extremely fresh in my mind.

Read more

Social Schedules: My First Facebook Application

I have had this idea for several years, I had always thought the way of sharing schedules on Facebook was inefficient. At my high school I would have to go through 1200 notes, just to see who was in my class.

My application, only available for MHS students, allows you to add your schedule (teachers and classes), see who’s in your class, comment and Facebook Like both your teachers and your hours.

It was written on the Google App engine, using an OO (object oriented) database with JSP’s and Core-servlets. I  will be writing a post about this stuff later.

For some who want to complain about how “bad” it is, I remind you this was about 8 days of work. However here is some of my attached code for it. Note: this is only SOME, not all.

socialschedulesSource [zip]