spring21 blog

[转贴]Scalability Best Practices: Lessons from eBay

fujohnwang — Thu, 29 May 2008 14:37:16 +0800

作者: fujohnwang 链接：http://spring21.javaeye.com/blog/198257 发表时间: 2008年05月29日

声明：本文系JavaEye网站发布的原创博客文章，未经作者书面许可，严禁任何网站转载本文，否则必将追究法律责任！

原文出处http://www.infoq.com/articles/ebay-scalability-best-practices

At eBay, one of the primary architectural forces we contend with every day is scalability. It colors and drives every architectural and design decision we make. With hundreds of millions of users worldwide, over two billion page views a day, and petabytes of data in our systems, this is not a choice - it is a necessity.

In a scalable architecture, resource usage should increase linearly (or better) with load, where load may be measured in user traffic, data volume, etc. Where performance is about the resource usage associated with a single unit of work, scalability is about how resource usage changes as units of work grow in number or size. Said another way, scalability is the shape of the price-performance curve, as opposed to its value at one point in that curve.

There are many facets to scalability - transactional, operational, development effort. In this article, I will outline several of the key best practices we have learned over time to scale the transactional throughput of a web-based system. Most of these best practices will be familiar to you. Some may not. All come from the collective experience of the people who develop and operate the eBay site.

Best Practice #1: Partition by Function

Whether you call it SOA, functional decomposition, or simply good engineering, related pieces of functionality belong together, while unrelated pieces of functionality belong apart. Further, the more decoupled that unrelated functionality can be, the more flexibility you will have to scale them independently of one another.

At the code level, we all do this all the time. JAR files, packages, bundles, etc., are all mechanisms we use to isolate and abstract one set of functionality from another.

At the application tier, eBay segments different functions into separate application pools. Selling functionality is served by one set of application servers, bidding by another, search by yet another. In total, we organize our roughly 16,000 application servers into 220 different pools. This allows us to scale each pool independently of one another, according to the demands and resource consumption of its function. It further allows us to isolate and rationalize resource dependencies - the selling pool only needs to talk to a relatively small subset of backend resources, for example.

At the database tier, we follow much the same approach. There is no single monolithic database at eBay. Instead there is a set of database hosts for user data, a set for item data, a set for purchase data, etc. - 1000 logical databases in all, on 400 physical hosts. Again, this approach allows us to scale the database infrastructure for each type of data independently of the others.

Best Practice #2: Split Horizontally

While functional partitioning gets us part of the way, by itself it is not sufficient for a fully scalable architecture. As decoupled as one function may be from another, the demands of a single functional area can and will outgrow any single system over time. Or, as we like to remind ourselves, "if you can't split it, you can't scale it." Within a particular functional area, then, we need to be able to break the workload down into manageable units, where each individual unit retains good price-performance. Here is where the horizontal split comes in.

At the application tier, where eBay's interactions are by design stateless, splitting horizontally is trivial. Use a standard load-balancer to route incoming traffic. Because all application servers are created equal and none retains any transactional state, any of them will do. If we need more processing power, we simply add more application servers.

The more challenging problem arises at the database tier, since data is stateful by definition. Here we split (or "shard") the data horizontally along its primary access path. User data, for example, is currently divided over 20 hosts, with each host containing 1/20 of the users. As our numbers of users grow, and as the data we store for each user grows, we add more hosts, and subdivide the users further. Again, we use the same approach for items, for purchases, for accounts, etc. Different use cases use different schemes for partitioning the data: some are based on a simple modulo of a key (item ids ending in 1 go to one host, those ending in 2 go to the next, etc.), some on a range of ids (0-1M, 1-2M, etc.), some on a lookup table, some on a combination of these strategies. Regardless of the details of the partitioning scheme, though, the general idea is that an infrastructure which supports partitioning and repartitioning of data will be far more scalable than one which does not.

Best Practice #3: Avoid Distributed Transactions

At this point, you may well be wondering how the practices of partitioning data functionally and horizontally jibe with transactional guarantees. After all, almost any interesting operation updates more than one type of entity - users and items come to mind immediately. The orthodox answer is well-known and well-understood - create a distributed transaction across the various resources, using two-phase commit to guarantee that all updates across all resources either occur or do not. Unfortunately, this pessimistic approach comes with substantial costs. Scaling, performance, and latency are adversely affected by the costs of coordination, which worsens geometrically as you increase the number of dependent resources and incoming clients. Availability is similarly limited by the requirement that all dependent resources are available. The pragmatic answer is to relax your transactional guarantees across unrelated systems.

It turns out that you can't have everything. In particular, guaranteeing immediate consistency across multiple systems or partitions is typically neither required nor possible. The CAP theorem, postulated almost 10 years ago by Inktomi's Eric Brewer, states that of three highly desirable properties of distributed systems - consistency (C), availability (A), and partition-tolerance (P) - you can only choose two at any one time. For a high-traffic web site, we have to choose partition-tolerance, since it is fundamental to scaling. For a 24x7 web site, we typically choose availability. So immediate consistency has to give way.

At eBay, we allow absolutely no client-side or distributed transactions of any kind - no two-phase commit. In certain well-defined situations, we will combine multiple statements on a single database into a single transactional operation. For the most part, however, individual statements are auto-committed. While this intentional relaxation of orthodox ACID properties does not guarantee immediate consistency everywhere, the reality is that most systems are available the vast majority of the time. Of course, we do employ various techniques to help the system reach eventual consistency: careful ordering of database operations, asynchronous recovery events, and reconciliation or settlement batches. We choose the technique according to the consistency demands of the particular use case.

The key takeaway here for architects and system designers is that consistency should not be viewed as an all or nothing proposition. Most real-world use cases simply do not require immediate consistency. Just as availability is not all or nothing, and we regularly trade it off against cost and other forces, similarly our job becomes tailoring the appropriate level of consistency guarantees to the requirements of a particular operation.

Best Practice #4: Decouple Functions Asynchronously

The next key element to scaling is the aggressive use of asynchrony. If component A calls component B synchronously, A and B are tightly coupled, and that coupled system has a single scalability characteristic -- to scale A, you must also scale B. Equally problematic is its effect on availability. Going back to Logic 101, if A implies B, then not-B implies not-A. In other words, if B is down then A is down. By contrast, if A and B integrate asynchronously, whether through a queue, multicast messaging, a batch process, or some other means, each can be scaled independently of the other. Moreover, A and B now have independent availability characteristics - A can continue to move forward even if B is down or distressed.

This principle can and should be applied up and down an infrastructure. Techniques like SEDA (Staged Event-Driven Architecture) can be used for asynchrony inside an individual component while retaining an easy-to-understand programming model. Between components, the principle is the same -- avoid synchronous coupling as much as possible. More often than not, the two components have no business talking directly to one another in any event. At every level, decomposing the processing into stages or phases, and connecting them up asynchronously, is critical to scaling.
Best Practice #5: Move Processing To Asynchronous Flows

Now that you have decoupled asynchronously, move as much processing as possible to the asynchronous side. In a system where replying rapidly to a request is critical, this can substantially reduce the latency experienced by the requestor. In a web site or trading system, it is worth it to trade off data or execution latency (how quickly we get everything done) for user latency (how quickly the user gets a response). Activity tracking, billing, settlement, and reporting are obvious examples of processing that belongs in the background. But often significant steps in processing of the primary use case can themselves be broken out to run asynchronously. Anything that can wait should wait.

Equally as important, but less often appreciated, is the fact that asynchrony can substantially reduce infrastructure cost. Performing operations synchronously forces you to scale your infrastructure for the peak load - it needs to handle the worst second of the worst day at that exact second. Moving expensive processing to asynchronous flows, though, allows you to scale your infrastructure for the average load instead of the peak. Instead of needing to process all requests immediately, the queue spreads the processing over time, and thereby dampens the peaks. The more spiky or variable the load on your system, the greater this advantage becomes.

Best Practice #6: Virtualize At All Levels

Virtualization and abstraction are everywhere, following the old computer science aphorism that the solution to every problem is another level of indirection. The operating system abstracts the hardware. The virtual machine in many modern languages abstracts the operating system. Object-relational mapping layers abstract the database. Load-balancers and virtual IPs abstract network endpoints. As we scale our infrastructure through partitioning by function and data, an additional level of virtualization of those partitions becomes critical.

At eBay, for example, we virtualize the database. Applications interact with a logical representation of a database, which is then mapped onto a particular physical machine and instance through configuration. Applications are similarly abstracted from the split routing logic, which assigns a particular record (say, that of user XYZ) to a particular partition. Both of these abstractions are implemented in our home-grown O/R layer. This allows the operations team to rebalance logical hosts between physical hosts, by separating them, consolidating them, or moving them -- all without touching application code.

We similarly virtualize the search engine. To retrieve search results, an aggregator component parallelizes queries over multiple partitions, and makes a highly partitioned search grid appear to clients as one logical index.

The motivation here is not only programmer convenience, but also operational flexibility. Hardware and software systems fail, and requests need to be re-routed. Components, machines, and partitions are added, moved, and removed. With judicious use of virtualization, higher levels of your infrastructure are blissfully unaware of these changes, and you are therefore free to make them. Virtualization makes scaling the infrastructure possible because it makes scaling manageable.

Best Practice #7: Cache Appropriately

The last component of scaling is the judicious use of caching. The specific recommendations here are less universal, because they tend to be highly dependent on the details of the use case. At the end of the day, the goal of an efficient caching system to maximize your cache hit ratio within your storage constraints, your requirements for availability, and your tolerance for staleness. It turns out that this balance can be surprisingly difficult to strike. Once struck, our experience has shown that it is also quite likely to change over time.

The most obvious opportunities for caching come with slow-changing, read-mostly data - metadata, configuration, and static data, for example. At eBay, we cache this type of data aggressively, and use a combination of pull and push approaches to keep the system reasonably in sync in the face of updates. Reducing repeated requests for the same data can and does make a substantial impact. More challenging is rapidly-changing, read-write data. For the most part, we intentionally sidestep these challenges at eBay. We have traditionally not done any caching of transient session data between requests. We similarly do not cache shared business objects, like item or user data, in the application layer. We are explicitly trading off the potential benefits of caching this data against availability and correctness. It should be noted that other sites do take different approaches, make different tradeoffs, and are also successful.

Not surprisingly, it is quite possible to have too much of a good thing. The more memory you allocate for caching, the less you have available to service individual requests. In an application layer which is often memory-constrained, this is a very real tradeoff. More importantly, though, once you have come to rely on your cache, and have taken the extremely tempting steps of downsizing the primary systems to handle just the cache misses, your infrastructure literally may not be able to survive without it. Once your primary systems can no longer directly handle your load, your site's availability now depends on 100% uptime of the cache - a potentially dangerous situation. Even something as routine as rebalancing, moving, or cold-starting the cache becomes problematic.

Done properly, a good caching system can bend your scaling curve below linear - subsequent requests retrieve data cheaply from cache rather than the relatively more expensive primary store. On the other hand, caching done poorly introduces substantial additional overhead and availability challenges. I have yet to see a system where there are not significant opportunities for caching. The key point, though, is to make sure your caching strategy is appropriate for your situation.

Summary

Scalability is sometimes called a "non-functional requirement," implying that it is unrelated to functionality, and strongly implying that it is less important. Nothing could be further from the truth. Rather, I would say, scalability is a prerequisite to functionality - a "priority-0" requirement, if ever there was one.

I hope that you find the descriptions of these best practices useful, and that they help you to think in a new way about your own systems, whatever their scale.

About the Author

Randy Shoup is a Distinguished Architect at eBay. Since 2004, he has been the primary architect for eBay's search infrastructure. Prior to eBay, he was the Chief Architect at Tumbleweed Communications, and has also held a variety of software development and architecture roles at Oracle and Informatica.

He presents regularly at industry conferences on scalability and architecture patterns.
本文的讨论也很精彩，浏览讨论>>

JavaEye推荐

闲来重拾毛笔...

fujohnwang — Sun, 18 May 2008 16:39:33 +0800

作者: fujohnwang 链接：http://spring21.javaeye.com/blog/194104 发表时间: 2008年05月18日

声明：本文系JavaEye网站发布的原创博客文章，未经作者书面许可，严禁任何网站转载本文，否则必将追究法律责任！

本文的讨论也很精彩，浏览讨论>>

JavaEye推荐

I like this smile~

fujohnwang — Fri, 09 May 2008 23:02:39 +0800

作者: fujohnwang 链接：http://spring21.javaeye.com/blog/191603 发表时间: 2008年05月09日

声明：本文系JavaEye网站发布的原创博客文章，未经作者书面许可，严禁任何网站转载本文，否则必将追究法律责任！

don't u think he is so damn cool,ha?

本文的讨论也很精彩，浏览讨论>>

JavaEye推荐

要严肃，还是要诙谐，这是个问题

fujohnwang — Thu, 17 Apr 2008 10:24:50 +0800

作者: fujohnwang 链接：http://spring21.javaeye.com/blog/183673 发表时间: 2008年04月17日

声明：本文系JavaEye网站发布的原创博客文章，未经作者书面许可，严禁任何网站转载本文，否则必将追究法律责任！

前阵子本来打算SprngMVC一章写完之后开始Spring Remoting的编写，为了能够让大家以亲近并且容易理解的方式来讲解Remoting，我是绞尽脑汁苦想了近乎两天，最后，起草了如下的这幅示意图：

当然这是第一版，初步搞定之后，我就发给a9看，a9比较赞同我这种表达方式，而且提出一些改进意见，等晚上碰到kruce，问他什么感性，这小子一句“靠，太不严肃了”，呵呵，整的我有些无言以对的感觉，不知道大家在一本书中如果看到这样的一副示意图，会做何感想那？！
本文的讨论也很精彩，浏览讨论>>

JavaEye推荐

回顾K-1经典-Mirko Cro Cop

fujohnwang — Sun, 13 Apr 2008 20:03:21 +0800

作者: fujohnwang 链接：http://spring21.javaeye.com/blog/182369 发表时间: 2008年04月13日

声明：本文系JavaEye网站发布的原创博客文章，未经作者书面许可，严禁任何网站转载本文，否则必将追究法律责任！

本文的讨论也很精彩，浏览讨论>>

JavaEye推荐

一副示意图的构思过程

fujohnwang — Tue, 25 Mar 2008 11:19:34 +0800

作者: fujohnwang 链接：http://spring21.javaeye.com/blog/175697 发表时间: 2008年03月25日

声明：本文系JavaEye网站发布的原创博客文章，未经作者书面许可，严禁任何网站转载本文，否则必将追究法律责任！

俗话说得好，"一图胜千言"(A picture is better than a thousand words), 当你看到一幅好的示意图的时候，可否想过要产生这么一副图片背后的艰辛？！呵呵，今天上午折腾了"一上午"才弄出最后觉得还算满意的图片...

初稿第一版和第二版构思

初稿第三版和第四版构思

最终决定

本文的讨论也很精彩，浏览讨论>>

JavaEye推荐

unveil spring book

fujohnwang — Fri, 26 Oct 2007 10:13:22 +0800

作者: fujohnwang 链接：http://spring21.javaeye.com/blog/135627 发表时间: 2007年10月26日

声明：本文系JavaEye网站发布的原创博客文章，未经作者书面许可，严禁任何网站转载本文，否则必将追究法律责任！

spring21.cn

几个月前开始有的念头，遂写下本书的第一章部分内容，书名暂且定为《揭密Spring(Unveil Spring)》，
鄙人将完成的书籍的ioc一章部分内容发布到该地址，切希望大家可以涉足一看，可以给出更多斧正意见，以便作出更多改进.
本文的讨论也很精彩，浏览讨论>>

JavaEye推荐

我在英极的日子

fujohnwang — Thu, 30 Aug 2007 11:25:31 +0800

作者: fujohnwang 链接：http://spring21.javaeye.com/blog/118325 发表时间: 2007年08月30日

声明：本文系JavaEye网站发布的原创博客文章，未经作者书面许可，严禁任何网站转载本文，否则必将追究法律责任！

Table of Contents

1. 英极编年史之事件篇
2. 英极编年史之个人篇

今天开玩笑说我的新书的前言写完了，邹晓旭问啥书？是不是《我在英级的日子》? 呵呵，想想也是哈，不过，这个题材要写本书有些单薄了，所以，写篇回忆录吧，OK, let‘s start！

1. 英极编年史之事件篇

2004年加入英极，直接进入credit项目组，一干就是2年多，基本上，从加入公司就开始了朝九晚九的生活，周末只休一天，周六算作正常上班，后来好像到2005年随着某些项目经理的提议，英极才将每周休息一天变为隔周休息一天，当然这些算是后话。刚进英极，credit算是当时公司最大的项目组了，充其量也不过10人左右，呵呵，到04年12月1号credit第一期发布成功，我们基本经历了三个阶段：朝九晚九，朝九晚11和黑白颠倒。不过这还算好的了，毕竟我们发布成功了，另一个组可不是那么幸运了。

当时除了credit，隔壁还有一个FX的项目组，他们当时的“拼命”程度比我们还狠，当然啦，一定程度上是被逼无奈，因为项目发布后就搞砸了，当时比较有意思的就是，FX第二天要发布的时候，所有的组员居然都不知道这一天是正式发布，甚至项目经理好像都不清楚，当然这种情况下发布出去的产品，不砸基本就是impossible了。至此坐在隔壁的我们就可以看到这样的一幅景象，FX的兄弟们基本就是以公司为家，困了找张被子往桌子底下一铺迷糊一会儿，醒了接着干，呵呵，最长好像有三天三夜连着干的，靠，简直tmd的超人那！相对于FX来说，credit前期虽然也是辛苦，但是在12月1号发布后，没有出现大的问题，算是一次成功的发布，所以没有搞得跟他们那么惨。

到2004年底，英极基本上就这两个大一点儿并且虎虎生风的项目组，其他mac机以及c之类的开发组，没有几个人，应该算维护一类的工作吧，2004年，基本上是在这两个项目组的工作中谢幕的。

进入2005年的时候，哦，可能是04年底的时候就开始了，随着日本活力门（livedoor）集团的崛起，作为其中国的子公司，英极也随之开始了扩张。英极也从原来几十人的公司扩张到了三四百人，当初英极最大的项目组credit，这个时候也只能干称小弟了，根本没法跟那时候三部张凌涛带领的200多人的证券组相比。不光国内英极这边开发迅速扩张，大连这边的对日服务的CallCenter也同样迅速扩编，极力吸引日本毕业生到大连这边工作。那个时候的英极也随着活力门集团而活力十足，魅力四射，不时的可以看到崭新的面孔，各项工作也随着部门的设立和规范而有条不紊的进行。 Credit虽然那个时候基本稳定了，人员上不像初期那个时候那么紧张，但还是吸纳了几个新人，老王那个时候已经是公司的CTO兼开发部部长，credit由老李做PM继续后继工作。

不过俗话说的好的，“三十年河东，三十年河西”，06年元旦左右，随着livedoor的传奇人物因为金融诈骗问题在日本那边受审，livedoor的神话也开始轰然破灭了，可谓树倒猢狲散，英极也无法单独承担原来的扩张所带来的压力，遂于某月周例会上宣布，打算几周后开始进行人员调整。先从那个组最先开刀我是记不清楚了，因为credit应该算是最晚的吧。反正你可以想想，或者早就应该有所耳闻，三部进行人员调整的时候，基本上是一片“苦爹喊娘”的景象，当然，我是不愿凑热闹看这些东西，不过听说保安啥都上了，应该有不少情绪控制不了的吧！从后来的红黑榜上骂娘的就可以知道这次“人员调整”造成的结果如何了。证券组当时虽然是最大的项目组，但到livedoor出事儿之后，跟某些bank的谈判一直没有搞定，所以，开发方面也是迟迟没有全面的开展，部分人员闲散严重。当每天上班就是上上网聊聊天的时候，突然说就被告知第二天不用来上班了，可以想想那些场景的出现也不足为奇了吧。不是我没有同情心，其实，“骂娘”只能说明你自己有问题，有那个时间“骂娘”，早干啥去了？！自身没有忧患意识，也怨不得任何人的！

反正三部裁员是大头，剩下的就是零零散散了，到我合同到期，老王安排我去天安的时候，人员从三四百好像弄到了200左右吧，当然，也可能我记错了，不过大体差不多。 credit当时还没有怎么动，不过，我到天安之后，老李也跑到了海辉，之后credit也开始了“人员变动”，呵呵，这些是我不在的时候发生的，具体啥情况我也就不想扯了...

总之，从时间跨度上来说，我经历了英极初期的艰辛到鼎盛时期的辉煌，然后部分见证了随着livedoor大厦轰然倒塌后英极随之的衰败，so this is the thing as it is.

2. 英极编年史之个人篇

我是2004年6月10日到英级的，我也不知道为啥这个日子记得那么清楚，或许到英级本来就够突然的吧！本来那个时候我在联系一些公司准备从常州国光辞职的，因为国光跟日立的项目洽谈一直没有什么进展，人一直闲着，所以，为了前途着想，不得不到处hunt新的职位，苏州上海等地的公司，最后敲定了一家上海的公司，开源产品比较偏重的，说好周六过去上海签约。但是，这个时候恰好原来做兼职时候的PM在msn上跟我提到大连这边英极也在招人，而且待遇不错，所以我就直接发了封email过来，说让这边2日内给我一个答复（周六要去上海签约嘛），没想到当天曲静就电话过来了，然后老王的电话面试，笔试是net上传给我的，大约半个小时搞定了吧，然后send回来，反正到现在我也不知答的咋样，呵呵。那天下午曲静就msn问啥时候能过去，说了一些啥6.10号恰好还有公司组织出去玩之类的诱惑之词，然后就让她给骗过来了，应该是6.9号的飞机，当晚在大学同学那里凑合一晚，第二天到公司报的道。

进公司第一天当然不免各种手续和流程啦，先是曲静哇啦哇啦一堆什么公司规章制度啥的，然后是王珏作为开发本部部长进来给我讲讲公司的情况，最后才是曲静说的“人很好的”即将成为我的项目经理的老王同志，进来也没扯啥，直接带我到开发组了，再细节的东西不记得了，最深刻的是当时看到都穿拖鞋，呵呵，这一点后来才明白。

刚进credit的时候，项目刚刚启动，还处于前期需求分析和技术调研阶段，记得当时组员也就那么几个人吧，现在都能数出来：老王(PM),徐敬琪(WhiteSock,袜子)，曲天连(肥猫，garfield)，大凯，凌爱国，杨小明，最后就是我和同样刚进项目组的老迟同志，呵呵。那个时候是大长桌子，几个人都在一个大的隔断里面，而且英极当时充其量也就几十个人吧，整个大厅其实还算满“宽敞”的。测试人员那个时候和开发不在一个隔断，所以，晓丽和阿梅在项目初期不是很熟。翻译当时是财哥，小金啥时候进来的我还真忘了，应该是04年末以后吧。随着项目人员的需求，老李，阿俊也陆续加入CREDIT，到2005年，基本上credit人员也凑齐了，贾维，阿九，晓强，鲁鹏...

刚进credit那会儿，对swt和jface之流根本就是一窍不通，因为之前一直做J2EE的web方面开发，所以，马上转到standalone的应用开发，多少有些不适应。反正只是记得老王直接把swt的新闻组网址往我的IE里面一敲，就放给我看了，靠，现在想想，还亏英文过得去，不然新闻组那些东西打死都搞不懂啊。除了SWT/JFace，还和袜子负责各种技术（hibernate之类）的调研，然后写调研报告...

当日方的式样讲解差不多的时候，基本就开干了，毫不隐讳的说，式样讲解我是大部分没有听懂，因为听听着就迷糊，呵呵，当然，那个时候毕竟经验有限嘛。这开干起来，那可就没个白天黑夜咯。基本上老王那时候属于强夜猫子型，白天他乱七八糟事情缠身，所以，干活儿没啥效率，那么就晚上发飙，通常是后半夜反而精神抖擞的。这可就害惨我们了，我从小属于长睡型，一睡觉少了就没精神，一过后半夜眼皮肯定打架，开始规定是晚上9点下班，后来因为12月1号credit第一期要发布，就直接天天11点才能下班回家，这个时候还算好的，起码能回家睡（当然，也不能叫回家睡啦，因为那个时候寄居在同学那里），每天班车最后肯定是剩下我，晓丽，还有到大纺的一个哥们，这哥们后来走了。到接近发布的前一个月，credit全员纯粹就颠倒了，晚上和后半夜干活儿，白天回家睡觉，现在我都不知道那个时候我是怎么熬过来的，听老王说是，一到后半夜我就不行了，呵呵

这种生活持续了半年多，天天如此，月月如此，所以，你也不难理解说有些人刚来英极1天，第二天就不来上班了，呵呵。试用期到了要转正的时候，我想过要闪人的，不过最后还是留下来了，具体为啥我也记不清楚了。反正，那时候杨小明已经走了，我应该是第二个老王找的谈的，最后就记得他那句“操，怎么都那么有想法”，呵呵，估计前面的人跟我的闪人想法都差不多，哈，最终留下来我估计我也没啥高尚的想法吧，呵呵，一个我贪心，多要了点儿工资，另一个就是看老王跟吉田较劲，我也有点儿要较劲的感觉，就好像你能顶，我咋就不能顶下去的意思，who knows，呵呵

credit第一期开发完成之后，第二期和后继开发虽然也是时有加班，但已经不像原来那么“惨烈”了，基本上每周三发布一次版本。从啥时候让我负责版本发布我记不清楚了，估计初期就已经给我了吧，InstallAnywhere从5.5的调研到吉田自己付费的6.0版本，最终发布文件的打包都是我这里做的。所以，到后期的时候，我基本上就没啥开发任务了，呵呵一到周三发布，如果手头有开发任务就对应一下，没有就上上网，吃吃零食（当时晚上加班有补助的，而且周三通常都要到第二天早上才能验证完毕）。但是，还是一样，我一到后半夜还是犯困，通常都是老王或者老李看测试通过了，走过来叫醒我，”阿福，开始打版本了“，我就睡眼惺松的爬起来，开始抖擞精神着手发布。迷糊归迷糊，不过打包制作installer直到上传服务器发布，好像我还没有因为马虎出过问题，幸哉幸哉，呵呵

那段因为晚上最早九点下班回家嘛，所以晚饭肯定不能回去吃了，所以，一段时间我们就三五成群的往黑石礁或者东财院里跑，冬天是跟老李，阿俊吃黑石礁路边摊的蒸饺最多，那家蒸饺做的比较有特色，价格便宜量又足，”我们一直用它“，呵呵，主要是咸菜还免费，冬天蒸饺加蒜泥，那叫一个爽啊。其他有段时间是老李，袜子，曲儿往东财院里儿跑，那里许多小饭店，价格不贵，吃的也不错，几乎每天晚上换一家或者同一家换不同的菜吃，呵呵，AA嘛，今天你掏了，明天就是我掏,回想起来那个时候还是挺爽的哈。

05年英极最鼎盛的时候，credit组我手头上的工作基本也没啥了，InstallAnywhere啥的也开始能transfer给刚进来的吴晓军都transfer给他，恰好公司准备搞全员的一个技术培训，包括spring，hibernate，linux系统以及各个数据库等内容。因为我是比较早在项目里面使用spring的，所以，顺便跟老王提了一句说spring那块儿由我来讲，后来就我和Roger配合进行的spring的培训，我主讲IoC和AOP，他负责Business Middle Tier以及MVC相关内容。培训总结在我之前的日志里面也有所总结，呵呵，这里就不扯了。

后期对credit项目做了一定的回顾，主要是技术架构方面的东西，在之前的blog中也有所阐述，毕竟技术和理念都是在发展的，即使你不能从原来的legacy系统转换过来，但是，所谓温故而知新，你可以从原来的系统中发现不足并进行改进。相对于那个时期我提出的总结，感觉还是比较恰当的，因为系统当时确实存在这些可以改进的问题。但是，任何观点或者方案的提出都是有其背景和前提条件的，如果不能满足这些，或许提出的改进方案反而会带来负面的结果。从看FES的front组使用EclipseRCP就可以说明以上观点，在没有相应技术储备，没有相应master的开发人员，在短时间内上线一个稳定的产品，还是存在一定风险的，这当然是后话，与英极没关系，呵呵

06年上半年在项目组工作不是很忙的情况下，我报了驾校，开始了考取驾照生涯。看起来好像请假挺多，但是，结束后回头看看，好像加起来也就三天假期，呵呵，自己都有些不相信，靠，没摸几天车就把驾照拿到手了。不过偶比较有自知之明，将来要上路肯定要先找陪练先提前跑跑，呵呵，总之那，一句话，偶06年拿到了偶人生中的第一本驾照，C照哦，哈哈

随着livedoor一帮元老与英极上层又重新组建了恒睿软件，偶合同到期后就被老王“连哄带骗(kidding)”的给弄到这边来了，英极的个人编年史就此结束。

本文的讨论也很精彩，浏览讨论>>

JavaEye推荐