Kent Langley is a Senior Director at Joyent, a cloud computing provider running and offering OpenSolaris Nevada based virtual servers, called Accelerators. Not only was he kind enough to answer my questions about his role at Joyent, details about the brand new Accelerator release, and Solaris, but he even brought in Ben Rockwood to give us an insight into the OpenSolaris community and Joyent’s reasons for using pkgsrc.
Kent, you joined Joyent a couple of months back. What are your responsibilities as Sr. Director?
My full title is Sr. Director of Scale Consulting. My focus is a blend of business and technical goals. At this time I work in and manage the Joyent Channel Partner programs, Joyent Labs, and Scale Consulting Services. But, one of the most important things I do here is leverage my experiences to help customers large and small to understand and meet the challenges faced when migrating to a Cloud Computing infrastructure. One of the great things here is that migrating customers to Joyent is relatively easy overall because of how much the overall technology stack mirrors what they are often already used to seeing in their own data centers.
Joyent is releasing a brand new version of its Accelerator. What are the major improvements for power users?
Yes 2.1.x will be releasing in just a few days. The amazing teams here at Joyent created the Accelerator and all the surrounding technology that enables it to do what it does so well. It is an embodiment of a huge amount of Systems, Software, and Internet knowledge.
The biggest improvements for most users out of the box will be things like integration of Phusion Passenger (mod_rails), updated versions of Java, PHP, Ruby on Rails, Python, Erlang, Perl, MySQL, Postgres. We actually, for the first time, will be including ffmpeg and mencoder as well which is interesting if you need them. Also, all the celebs will be at the 2.1.x release party. Apache, memcached, varnish on Solaris, Merb, MySQL, Postfix, Mono, Subversion, Git, and a whole lot more.
Why did you switch from Blastwave to pkgsrc. And why did you completely remove IPS?
I’ll roughly quote Ben Rockwood here. He says we did it for “fine grained control over versions of the packages that we need to support and use for ourselves and our clients. Also, Blastwave seems to have some stability issues.”
We didn’t really remove IPS. It’s just not part of Nevada. So, it wasn’t there to remove in the first place.
The switch from blastwave to pkgsrc was official in March 2008. One of the things that we have been working hard to do is make things generally easier for people developing and deploying internet applications in the cloud that want the power of Solaris, Dtrace, and ZFS but might be more familiar with other operating systems. Using pkgsrc allows us to take a step in this direction. Additionally, we maintain a significant pkgsrc repository for use by our clients. Recently a beta client said, “The new template is so fully configured that pretty much everything that had been causing me issues is now just installed by default.” This is probably true for anyone that uses or ever used Solaris for anything. It’s kind of a more friendly Solaris. An accelerator is a full featured building block for creating much more complex services while reducing operational efforts and TCO substantially.
You’re currently running snv_89. OpenSolaris 2008.05 was snv_86 and current (as of this writing) is snv_99. How do you decide which build is stable enough for you to deploy? How much different is your version of OpenSolaris from the official releases? What do you tweak (besides using pkgsrc instead of IPS)?
Technically we don’t use OpenSolaris. The naming conventions are unfortunate. The best way to think of it is that we do not use Indiana. We run the Nevada build. Again, to quote Ben Rockwood, “we do extensive testing of each release and determine which is best and most importantly stable.” Stability is a primary concern, not bells and whistles.
You aren’t using zfs as root filesystem for your Accelerator zones. Why not? Are there any serious technical issues holding you back or is it more a matter of taste?
Actually, we do use ZFS for all the filesystems here. It’s just that most of the details of that are hidden at the Accelerator level.
You offer shared (Managed) Big IP f5 load balancers to be used to, well, load balance your customers applications. What type of access and configuration possibilities do customers have themselves? When would you recommend starting to use Big IPs?
Direct access to the BigIP’s by customers is not allowed. It’s essentially a managed service. We will work with clients to create and install custom IRules, VIP’s, and any variety of setups. This is an extremely powerful part of our overall infrastructure.
A short list of the times to use the bigIP’s is when your traffic patterns have either exceeded the abilities of a single Acclerator at the application or even the database level in some cases, when you want to use IRules to do application switching based functional partitioning, when you want to “hide” the real IP address of your machine from the world, to replace a software based load balancer that is no longer able to keep up with demand, or if you want to build in some High Availability. IRules are extremely powerful.
You send out passwords via e-mail. You ask your users to secure their Accelerator in your welcome mail, but there are quite some steps to be done. I know of hosting companies sending out passwords to mobile phones only. What are the reasons for your current approach. Do you think about changing it? Couldn’t you e.g. offer (at least optionally) the possibility to upload a public key while ordering a new Accelerator?
Once your accelerator is delivered, it’s security becomes your responsibility at the password level. If you change those passwords we can’t even get into your Accelerators without some significant effort.
I think the concept of adding public key at order time would be nice. I’ll float that to the team.
You are running all possible kinds of web application stacks on your Accelerators. E.g. you talked about a huge drupal site on your blog and joyent is known to host high traffic ruby sites. What are your personal preferences and why?
Yes, the Accelerator is a very flexible building block. A short list of some stacks would include Tomcat, Gigaspaces XAP, Axiom Stack, GlassFish, CakePHP, Symfony, Drupal, Django, Merb, Ruby on Rails, Mono, CouchDB, Ejabberd, and SunGridEngine.
We host some of the largest Ruby on Rails, PHP, and other sites on the internet using Joyent Accelerators. There are multiple sites that serve billions of page views per month. These are no longer uncommon. Around here I joke that a billion (pageviews) is the new million.
I am pleased that our overall technology stack works well enough to do this even with existing frameworks. But, it’s rather brute force in many cases. For my personal preferences, I find most technology stacks and frameworks lacking for today’s needs when peak loads can be 10x or more above the mean.
Cloud Computing, the actual act of doing computing in the cloud, has very strong parallels (no pun intended) to distributed programming models. But, there are only a fraction of existing web frameworks that do this well or even at all. The more interesting ones allow or provide fully distributed access to a horizontally scalable data layer either in memory or on disk. It’s this data layer that’s changing things because it allows a path to scalability that’s been unavailable before and all to uncommon still today in every day use.
So, my personal preferences lean toward languages, frameworks, and products that follow the number of tenants – recently well put in an article in InfoQ about Ebay’s architectural policies.
http://www.infoq.com/articles/ebay-scalability-best-practices
There are claims that the OpenSolaris community is heavily dominated by Sun. What are your experiences with the OpenSolaris community?
I leaned on our resident Sun Community expert, Ben Rockwood, again for this answer:
The OpenSolaris Community is heavily dominated by Sun. The OpenSolaris Community took an existing close-sourced OS with an enormous corporate infrastructure comprised of several thousand people and opened it to all. It is natural that the community is dominated by Sun given that those people have been working on the code for the last 20 years are from Sun. However, a large amount of non-Sun community involvement exists. There are well over 100,000 registered community members (more than 3 times the size of Sun’s employee base) and growing steadily. Several non-Sun community members have served on the OpenSolaris Governing Board, including Ben Rockwood of Joyent. The CDDL was reviewed by Sun and non-Sun community members alike prior to its ISO launch, pre-OpenSolaris. Given that Solaris is the cornerstone of Sun Microsystems product it’s likely that the community will always be dominated by Sun, but its important to distinguish between “comprising the bulk of…” from “iron fisted control”.
What is your personal favorite feature in Solaris? Would you recommend using it? And why?
Broadly speaking there are two high profile features that make Solaris shine. One is DTrace and the other is ZFS. Once you begin to understand the power and flexibility of real time active tracing using DTrace it’s hard to make do with other tools.
I would totally recommend working with Solaris. However, I will be the first to admit that there is a learning curve. It’s not Linux, It’s not BSD. It’s Solaris and that comes in various flavors. As noted, our flavor of Solaris, the Accelerator, is different than most and more than likely a more gentle introduction without losing any of the great features.
Thanks a lot for taking the time to answer these questions.
My Pleasure. Thank you for being one of our best beta testers this round!
Agile Web Operations 