Derek Liu, Gaia Online, the one year later interview
A year ago Big Boards was interviewing Derek Liu, the administrator of Gaia Online, which being only a few months old already featured millions of posts. A year later, the board has grown into the largest forum on the web, with close to 200 million posts, and more than 2000 people online at any time. Derek has kindly accepted to tell us about this past year’s changes, and the upcoming ones.
Visit Gaia OnlineHello again Derek. When you first answered our questions one year ago, Gaia Online was 6 months old, had about 5 million posts and 100000 members. Today, your forum has received more than 180 million posts and is close to a million members. The first question will be, are you happy with this growth? Surprised?
Gaia’s growth rate was nothing we anticipated. 
We were expecting two or three times the popularity of other forums which I’ve hosted in the past, but the figures proved us wrong just a month after the site opened. We’ve never felt this happy to be proven wrong.
Plenty of new features have been implemented in a year. Gaians can now visit new shops, get a haircut in the hair salon, there is an auction system, a bank, users can create journals, guilds or enter a matchmaking area… Ultimately, what is your goal with Gaia ? What would you like it to be that it is not already?
Gaia’s mission statement remained the same - to be a free and fun hangout for everyone. To fulfill our mission is our goal. If we sit back one day and figured that “this idea would be cool”, we would put it on the whiteboard and wait for the chance to realize the idea. A lot of our upcoming ideas are related to games and quests, so things will start to take a different turn than the community related features that we’ve been implementing. A year ago we mentioned that Gaia will be more like a game than a forum, finally that statement will be realized in the coming months.
What is your target audience and what are its main characteristics? What aspects of your site is it the most responsive to?
We’ve recently hosted a poll which told us almost all our audiences are within the 14 to 25 year old age range. We have very strong interest groups in different forums, though just as much of our members are interested in the sheer ability to collect and customize their avatars. Of course, both aspect must play off each other. There’s little sense in building your own avatar if your avatar isn’t used to communicate to anyone else. 
How do you and your moderators keep up with such a large community? How do you take into account user input, monitor spam, and in the meantime create a forum culture?
We owe a lot to our current moderators. Between the 90 or so moderators and staff members we help take care of numerous forum moderation tasks. Moderators also help compile user input, report bugs, and within the moderators is a community within itself which focusses on how we can improve Gaia. Soon, a new member of the development team will be responsible for making features for our moderators as a first priority. In general we stick with providing our users the freedom of expression, and only to moderate when it’s necessary. Variety is the culture we strive for, and to maintain quality while providing freedom is the challenge we all battle with.
“Gaia’s mission statement remained the same - to be a free and fun hangout for everyone.”
Where do you look for inspiration? What websites do you find particularly innovative? Or do you prefer to search for ideas offline?
I believe a creator’s inspiration is only limited by the person’s experiences on-line and off-line, so there’s no limit as to where we perform our research. For interface ideas we look at things that are esthetically pleasing, such as video games, RPG in particular. For community ideas we look at all the popular web services and try to harvest features which will proove useful to Gaia members.
Last year you were searching for new developers to get involved in the project. How many members does your team contain now, what are their respective roles, and how do you manage them?
We were extremely fortunate to have found friends, referrals and volunteers that are dedicating their time as developers. Today there are about 10 of us handling the duties of hardware planning and installation, server software upgrades and security, software development from new features, sustaining features, bug fixes, performance enhancements, to database auditing. Aside from engineering, we also have our art team, flash programmers, member support, moderator support, and accounting plus administration. That’s definitely more jobs than what 10 people can handle. All of us wear different hats to get the job done, and everyone’s opinions are crutial to the development of the site. We’re all very experienced and competent in what we do, so I don’t need to manage anyone per say. The only thing which I need to manage is the direction of the site as a whole, which involves the maintenance and release of all the features and co-ordination between team members. Even that aspect is handled as a team. It’s an easy job when everyone love the site and we all get along plus we all love what we do.
How many servers are there now running Gaia? What kind of architecture have you implemented? What were the main difficulties you had in implementing it?
There are roughly 50 servers behind Gaia this month. Almost all servers are dual CPU servers since we used to have a limitation in space with a cabinet hosting solution. Though recently we switched over to co-locating inside a cage, and we’ll be implementing single CPU servers once again, which will increase our server count significantly in the near future while keeping costs low.
Gaia’s biggest growing pain is in changing our database abstraction layer which allows us to easily work with different database servers. It requires tremendous changes and bug fixing. It’s been 3 months since we implemented the change and we’re still fixing bugs related to these changes.
Regarding the current architecture, we currently have a really segregated setup with a load balancer in front of 30+ web servers, and two redundent firewalls in-front of the load balancer. A backend private network connects the web servers our databases. There is one cluster of 4 databases for handling the forum, and 7 other databases which handle specific tasks. On top of that we have our test server, staging server, memory cache server, 5 graphic servers, and about 10 web servers for handling specific features such as avatar creation. It’s like a boy collecting stamps, we collect a few servers every few month and today we ended up with a collection. One server is currently dedicated to archiving the forum content, while every database server performs their back-up respectively and very soon we’ll be installing a storage area network to hold all the data in one place for easier backup.
All database servers are still the dual Opteron configuration with 4GB memory, while the web server config is being changed from dual Xeon with 2 Gig memory to P4 servers with 1 Gig memory.
Some developers practicing other programming languages often describe PHP as less scalable than other alternatives. However, there are examples like Friendster and its 9 million members which recently switched from Java to PHP and obtained impressive results. Having the experience of managing a high traffic site built in PHP, what would be your take on this? What do you feel the bottleneck is in your architecture?
My personal opinion is never adequate I’m sure, but my focus would be how does one language affect the development cycle compared with another. Do we want Java, which requires more development time for features and less development time for scalability, or vice versa? In an environment where the timeline is tight and resulting codes can be scrapped or re-written because of user response, I believe that PHP suits us better. Not to mention that we couldn’t run anything but Linux plays a huge role in our decision making.
In the realm of performance and scalability, I’m positive that many programming oriented articles will be much more informative than what my puny programming experience can provide. We stick with the basic principles of seperating services into their respective databases and we strive to expand by adding more machines, not by upgrading one main server. That also means that for any developer, in the planning stage we should keep in mind that every feature should rest in their own logical database so that when time to expand comes, you can move the database into a physically different server with minimal code change. Also, web servers are easily scalable while scaling database servers is much harder, so we try to limit the use of features within MySQL and leave all the computation to PHP. Over the years I’ve noticed that one bad query can put a database to its knees. A performance audit with report of how long each query takes to run is very essential before the release of any piece of code.
“We are lucky to receive such unprecedented support from our members to help our site grow.”
The keyword search feature is currently disabled on Gaia. Do you plan on investigating new techniques, or have you given up on search?
We have several ideas for having a search feature, but because our site is very chat oriented, searching isn’t as frequently requested a feature compared to things like gaming related features. If and when the time comes, a database server will be dedicated towards searching for recent messages. Being able to search through archived messages might be too much, and we’ll probably pass that on to search engines such as google.
As your site grows, How is your work evolving? Are you still mostly involved in technical improvements, are you focussing on features, are the business aspects becoming more important?
A year ago, 65% of my time was used towards site development. These days I believe only 25% of my time is available for this task. Aside from overseeing all operation Gaia involves, communicating with people outside of our team is also vital. Putting on top a more active bug list, more user related issues, more time for meetings, more hardware failures, and a much bigger to-do list, my goal of putting a smile on everyone’s face becomes so daunting that I end up with a frown on my face when I’m stressed out.
Handling the growth and new features also means new investments. How are you monetizing your site, and how successful are you financially?
We are lucky to receive such unprecedented support from our members to help our site grow. We would not be able to keep up with the cost for bandwidth and hardware otherwise. Any extra income is used in consulting talents for adding things like flash games or other independent projects. We spend every dollar on site development, and although more income might lead to faster development for Gaia, we’re very happy and excited about the growth and our current development schedule.
Gaia’s home page still states it’s in alpha stages. When is the beta expected for? Looking at the Gaia map can let imagine an expanding world, more and more oriented towards roleplaying. Can you reveal us some of the next attractions to become available in Gaia?
Beta should be here when we put a code freeze on all community related features and stability become our first priority. That should be around the time half way into the development of our gaming related features. Soon all Gaia members will be able to buy and customize their own houses, venture into dungeons or engage in battles as an individual or as groups big or small. Earning gaia gold on Gaia will not be limited only to posting. As time spent on Gaia will not only earn you gaia online gold, but also earn you experience points that will affect other areas of the site. Much effort had been placed months ago on the features mentioned, and we can hardly wait until we introduce them to the community.
Tags: gaia gold gaia online gold