Dumb Coder

NetBeans Tips

2011-11-29T04:28:00.001-08:00

Use Eclipse Shortcuts

Tools -> Options -> Keymap -> select Eclipse profile

Create code template

Tools -> Options -> Editor -> "Code Templates"

Press tab to activate define code template.

Reset the layout of windows

Windows -> Reset windows

Float/Dock views

Export/Import Settings

Open Options Dialog: Tools > Options

Click Export/Import button

In NetBeans, we can run a project, run a file, or run a test file.

How to pass in command line arguments while using "Run File" or "Test File"

Select project -> Properties -> Choose Run node -> Type command line arguments in the Arguments text box.

Remote debug

Settings in Netbeans:

Debug -> Attach Debugger

In the "Attach" dialog:

Debugger: JPDA

Connector: Socket Attach

Transport: dt_socket

Enter host and port

Settings in server:

set JAVA_OPTS=%JAVA_OPTS% -Xrunjdwp:transport=dt_socket,address=9999,server=y,suspend=n

Update Sites

Additional Development Plugins

http://updates.netbeans.org/netbeans/updates/dev/uc/final/main/catalog.xml.gz

Latest Development Build

http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastStableBuild/artifact/nbbuild/nbms/updates.xml.gz

Static Development UC

http://updates.netbeans.org/netbeans/updates/dev/uc/final/main/catalog.xml

Dynamic Development UC

http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastStableBuild/artifact/nbbuild/nbms/updates.xml.gz

nbpluginportal

http://plugins.netbeans.org/nbpluginportal/files/nbms/pluginportal-update-center.xml

Netbeans logs

Windows XP: C:\Documents and Settings\joe\.netbeans\6.7\var\log\message.log

Linux: /home/joe/.netbeans/6.7/var/log/messages.log

Mac OS X: /Users/joe/.netbeans/6.7/var/log/messages.log

JBoss Tips

2011-11-28T04:24:00.000-08:00

Commands

shutdown.sh -S

run.sh -b 0.0.0.0 -c default

run.bat –Djboss.server.log.dir=d:/log

Change ports

In jboss 5, in $JBoss\server\default\conf\bindingservice.beans\META-INF\bindings-jboss-beans.xml, change the port offset of PortsDefaultBindings.

<bean name="PortsDefaultBindings"  class="org.jboss.services.binding.impl.ServiceBindingSet">
  <constructor>
     <!-- The port offset -->
     <parameter>1000</parameter>
     <!-- Set of bindings to which the "offset by X" approach can't be applied -->
     <parameter><null/></parameter>
  </constructor>
</bean>

To use the port assignments defined by the Ports01Bindings bean, set the jboss.service.binding.set system property to ports-01 when starting the application server.

run -c myconfig2 -Djboss.service.binding.set=ports-01

Configuring logging

The log4j configuration file is located at server/xxx/conf/jboss-log4j.xml.

Defining a rolling log appender

<appender name="FILE">
    <param name="Append" value="true"/>
    <param name="MaxFileSize" value="10MB"/>
    <param name="MaxBackupIndex" value="20"/>
    <param name="Threshold" value="ERROR"/>
</appender>

Defining logging for your application

You can log your application by adding category entries to the jboss-log4j.xml file.

<appender name="JBIA" ...>
    <param name="File" value="${jboss.server.log.dir}/jbia.log"/>
</appender>
<category name="org.jbia">
    <priority value="DEBUG"/>
    <appender-ref ref="JBIA" />
</category>

Enable http access log

Edit the server.xml file. For the default server in the default location, it is ${jboss.home}\server\default\deploy\jbossweb.sar\server.xml.

Uncomment the "Access logger" section. The log file will show up in server/default/log/localhost_access_log.{todays_date}.log

<Valve className="org.apache.catalina.valves.AccessLogValve"
                prefix="localhost_access_log." suffix=".log"
                pattern="common" directory="${jboss.server.log.dir}" 
                resolveHosts="false" />

Configuring startup parameters

Rename run.bat for example to run-old.bat.

Create another run.bat file and put this line to it:

set JAVA_OPTS=%JAVA_OPTS% -Xms512m -Xmx1024m -XX:MaxPermSize=512m

run-old.bat -b 0.0.0.0 -c default

Dump SOAP Messages

For the case that is not easy to model as a simple DFS request, it is useful to enable the SOAP dump.

For client side, set "com.sun.xml.ws.transport.http.client.HttpTransportPipe.dump=true";

For server side, set "com.sun.xml.ws.transport.http.HttpAdapter.dump=true";

In Jboass, put the following line in newly created run.bat:

set JAVA_OPTS=%JAVA_OPTS% -Dcom.sun.xml.ws.transport.http.HttpAdapter.dump=true

Enable remote debug

In newly created run.bat:

rem set JAVA_OPTS=%JAVA_OPTS% -Xrunjdwp:transport=dt_socket,address=9999,server=y,suspend=n

Load balancer with Apache

Configure apache httpd.conf:

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
<Proxy balancer://mycluster>
Order deny,allow
Allow from all
BalancerMember http://node1.mycluster.lan:8080/ route=node1 timeout=15
BalancerMember http://node2.mycluster.lan:8080/ route=node2 timeout=15
ProxySet stickysession=JSESSIONID|jsessionid
# other values: bybusyness|byrequests|bytraffic
ProxySet lbmethod=bytraffic 
</Proxy>
ProxyPass /jmx-console balancer://mycluster/jmx-console 
ProxyPassReverse /jmx-console  http://localhost:9080/jmx-console stickysession=JSESSIONID|jsessionid

Configure JBoss

In jboss, modify {jboss.home}\server\default\deploy\jbossweb.sar\server.xml to configure the jvmRoute for sticky sessions:

<Engine name="jboss.web" defaultHost="localhost" jvmRoute="node1">

Modify {jboss.home}\server\default\deployers\jbossweb.deployer\META-INF\war-deployers-jboss-beans.xml:

<bean name="WebAppClusteringDefaultsDeploye" class="org.jboss.web.tomcat.service.deployers.ClusteringDefaultsDeployer">
  <property name="useJK">true</property>
</bean>

Resources:

http://coldshen.com/blog/index.cfm/2008/9/27/Load-Balancing-your-JBoss-AS-5-cluster-with-Apache-AJP

Metro Logging

JBoss in Action: Configuring the JBoss Application Server

Useful Notepad++ Tips

2011-11-22T19:22:00.001-08:00

Three years ago, after used Notepad++ after some time, I write one article about NotePad++. Three years later, it's time to give it some update.

Common Settings

Open "Settings" -> "Preferences...".

In "Tab bar" section of General tab, select "Double click to close document" and "Multi-line".

In "Backup/Auto-Completion" tab, enable auto-completion option, and choose "Word completion".

Enable spell-checker

Install Aspell, afterwards enable spell-checker :)

Use "Ctrl+Tab"

Use "Ctrl+Tab" to switch between to you last viewed file, and quickly jump to the file you want.

Useful Plugins:

How to install plugins:

Go to "Plugins" -> "Plugin Manager" -> "Show Plugin Manager", this will bring up the "Plugin Manager" dialog where you can find many available plugins.

File switcher

Usually I opened many files in Notepad++, 20 - 30+, some files I need edit often, or some files I just forget to close. So sometimes, it would be hard to find files I want to edit in Notepad++.

File switcher is just the plugin that we need- It presents a sortable list, and we can type text to filter and find the file we want.

Compare Plugin - show the difference between 2 files.

XML Tools - XML Checker. Can auto-close tags and provides other useful features.

To Format XML:

Go to "Plugins" -> "XML Tools" -> "Pretty print (XML only -with line breaks) - Ctrl+Alt+Shift+B"

Explorer Plugin – a file browser

Speech Plugin

There are many other useful plugins that may meet your needs, please explore them.

Resources:

Is there a way to manage tabs efficiently in Notepad++

https://plus.google.com/101157854606139706613/posts/AePsc279Rs6

Link With Editor in Eclipse

2011-11-22T05:14:00.001-08:00

I remembered that in the old versions of eclipse(maybe 3.1), in the package and project explorer, it will show the current java class that is opened in the editor. When I change to another java class, it the explorer will automatically scroll to display the new class.

It seems this feature is gone in later release.

But the fact is that it is hidden - by default disabled, to enable it, you just need toggle the"Link with Editor" button in Package/Project/ or Navigator toolbar.

Resources:

Best Eclipse Feature: Link With Editor

Use Working Set to group projects in Eclipse

2011-11-22T05:04:00.000-08:00

Use Working Set to organize and group related projects visually in Eclipse

Usually, we develop or maintain different projects at same time, each project

has multiple modules, or related projects, such as test or demo projects.

We want to organize related projects visually, so in eclipse, it would look like:

projecta

---projecta-module1

---projecta-module2

---projecta-module3

---projecta-other-modules

projectb

---projectb--related-modules

To achieve this in Eclipse:

Create one working set, add related projects into it, and then (important step here), in project(or package) explorer, click the top down button -> "Top level elements" -> choose "Working Sets".

Resources:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=362968

Fail to start JBoss5 - Common unexpected

2011-10-03T03:32:00.001-07:00

Unable to start JBoss 5 - \Common was unexpected at this time

--Don't use quotes in path variable

Today, I download JBoss 5 in my Windows 7 - 64 bit machine, and try to use run.bat to start the server, but it fails, and reports error:

\Common was unexpected at this time.

Google searched, but found nothing useful.

So I tried to take a look at the run.bat, first I comment the first row "@echo off", so I will now where it stops with the error above. I located the source of the errors:

if not "x%JBOSS_NATIVE_HOME%" == "x" (

set "PATH=%JBOSS_NATIVE_HOME%;%PATH%;%JBOSS_HOME%\bin"

set JAVA_OPTS=%JAVA_OPTS% "-Djava.library.path=%JBOSS_NATIVE_HOME%;%PATH%;%JBOSS_HOME%\bin"

)

Then I check the PATH environment:

%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;"C:\Program Files (x86)\Common Files\EMC";C:\Program Files\Perforce;%JAVA_HOME%\bin;%ANT_HOME%\bin;%MVN_HOME%\bin

Yeah, find out the culprit: the quotes around the path to EMC common files - "C:\Program Files (x86)\Common Files\EMC"

After removed the quotes, start a new command line, JBoss can be started with no problem.

Lesson learned: don't put quotes in your environment variables, such as PATH, JAVA_HOME etc, this may make scripts broken. This also applies when installation program update these variables.

Interview Questions – Databases

2011-06-08T06:43:00.000-07:00

Interview Questions – Databases

Why Normalize?

1. Reduce Redundancy

I. Redundancy would waste more space.

One obvious drawback of data repetition is that it consumes more space and resources than is necessary.

II. Redundancy may cause inconsistency.

Moreover, redundancy introduces the possibility for error, and exposes us to inconsistencies whenever the data is not maintained in the same way. for example, these redundant data are not updated at same time.

2. Prevent insert, delete, update anomaly.

Insert Anomaly

The insert anomaly refers to a situation wherein one cannot insert a new row into a relation because of an artificial dependency on another relation.

Delete Anomaly

It refers to a situation wherein a deletion of data about one particular entity causes unintended loss of data that characterizes another entity.

Update Anomaly

The update anomaly refers to a situation where an update of a single data value requires multiple rows of data to be updated.

Example: Product, Customer, Invoice.

First Normal Form: Eliminating Repeating Data

A relation is said to be in first normal form when it contains no multivalued attributes.

To transform unnormalized relations into first normal form, we must move multivalued attributes and repeating groups to new relations

Second Normal Form: Eliminating Partial Dependencies

Arelationissaidtobein second normal formif itmeets both the following criteria:

• The relation is in first normal form.

• All non-key attributes are functionally dependent on the entire primary key.

Second normal form only applies to relations where we have concatenated primary keys.

Once we find a second normal form violation, the solution is to move the attribute(s) that is (are) partially dependent to a new relation where it depends on the entire key instead of part of the key.

Attribute B is functionally dependent on attribute A if at any moment in time, there is no more than one value of attribute B associated with a given value of attribute A.

In another word, it means A determines attribute B, or that A is a determinant (unique identifier) of attribute B.

In the INVOICE relation,we can easily see thatCustomerNumber is functionally dependent on Invoice Number because at any point in time, there can be only one value of Customer Number associated with a given value of Invoice Number.

Anti-example

INVOICE LINE ITEM: # Invoice Number, # Product Number, Product Description(only dependent on product number), Quantity, Unit Price, Extended Amount

Third Normal Form: Eliminating Transitive Dependencies

Arelationissaidtobein third normal formif itmeets both the following criteria:

• The relation is in second normal form.

• There is no transitive dependence (that is, all the non-key attributes depend only on the primary key).

To transforma second normal formrelation into third normal form, simplymove any transitively dependent attributes to relations where they depend only on the primary key. Be careful to leave the attribute on which they depend in the original relation as a foreign key.Youwill need it to reconstruct the original user viewvia a join.

An attribute that depends on another attribute that is not the primary key of the relation is said to be transitively dependent.

Anti-example

INVOICE: # Invoice Number, Customer Number, Customer Name(dependent on primary key attribute: Customer Number), Customer Address..., Terms, Ship Via, Order Date, Total Order Amount

The third normal form violation—a non-key attribute determining another non-key attribute

Boyce-Codd Normal Form - BCNF Form

Boyce-Codd normal form (BCNF) is a stronger version of third normal form. It addresses anomalies that occur when a non-key attribute is a determinant of an attribute that is part of the primary key (that is, when an attribute that is part of the primary key is functionally dependent on a non-key attribute).

The Boyce-Codd normal form has two requirements:

• The relation must be in third normal form.

• No determinants exist that are not either the primary key or a candidate key for the table. That is, a non-key attribute may not uniquely identify (determine) any other attribute, including one that participates in the primary key.

The solution is to split the unwanted determinant to a different table.

Anti-example

#Customer Number, #Product Line, Support Specialist Number

Fourth Normal Form

An additional anomaly surfaces when two or more multivalued attributes are included in the same relation.

Denormalization

Normalization leads tomore relations,which translates tomore tables and more joins.When database users suffer performance problems that cannot be resolved by other means, then denormalization may be required. Most database experts consider denormalization a last resort, if not an act of desperation.

Possible denormalization steps include the following:

• Recombining relations that were split to satisfy normalization rules

• Storing redundant data in tables

• Storing summarized data in tables

What Is a Transaction?

A transaction is a unit of work that is composed of a discrete series of actions that must be either completely processed or not processed at all.

Transactions have properties ACID (Atomicity, Consistency, Isolation, Durability).

• Atomicity

A transaction must remain whole. That is, it must completely succeed or completely fail. When it succeeds, all changes that were made by the transaction must be preserved by the system. Should a transaction fail, all changes that were made by it must be completely undone.

• Consistency

A transaction should transform the database from one consistent state to another consistent state.

• Isolation

Each transaction should carry out its work independent of any other transaction that might occur at the same time.

• Durability

Changes made by completed transactions should remain permanent

Read Committed Isolation Level

Read Committed is the default isolation level in PostgreSQL. When a transaction runs on this isolation level, a SELECT query sees only data committed before the query began; it never sees either uncommitted data or changes committed during query execution by concurrent transactions.

In effect, a SELECT query sees a snapshot of the database as of the instant that that query begins to run. Notice that two successive SELECT commands can see different data, even though they are within a single transaction, if other transactions commit changes during execution of the first SELECT.

Serializable Isolation Level

The level Serializable provides the strictest transaction isolation. This level emulates serial transaction execution, as if transactions had been executed one after another, serially, rather than concurrently.

In REPEATABLE READ, one transaction will not read committed change of another transaction.

set session transaction isolation level serializable;

Client 1:

mysql> start transaction;

mysql> select * from tt;

CLIENT 2:

mysql> update tt set d=2 where d=4;

ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

This time, client 2 cannot update the row. After a few seconds of waiting, the attempt to update a row ends with the error message “Lock wait timeout exceeded”.

Both REPEATABLE READ and SERIALIZABLE don’t allow dirty reads, nonrepeatable reads and phantom reads to happen.

While REPEATABLE READ still allows another client to modify data, while he performs a transaction. The change made by concurrent transaction would be invisible to current transaction.

SERIALIZABLE strictly disallows other transaction change data current transaction operates on.

Serializable vs. Snapshot

They are SERIALIZABLE and SNAPSHOT. They are both made available in order to avoid dirty, non-repeatable or phantom reads, but they do so using different methods.

SERIALIZABLE

The SERIALIZABLE level prevents phantom reads by using range locks.

SERIALIZABLE transactions take range locks in order to prevent Phantom Reads.

SNAPSHOT

While SERIALIZABLE uses locks, instead SNAPSHOT uses a copy of committed data. Since no locks are taken, when subsequent changes are made by concurrent transactions, those changes are allowed and not blocked.

If you try to make a change to some data and that data has already been changed by concurrent transaction, you will get update conflict error message.

Types of Indexes

http://it.toolbox.com/blogs/programming-life/sql-performance-get-the-right-index-7996

http://blog.csdn.net/laodao1/archive/2009/10/21/4709516.aspx

http://zhidao.baidu.com/question/87078855.html

Non-unique indexes

A non-unique index is one in which any key value may occur multiple times. This type of index is defined with the keyword INDEX or KEY.

ALTER TABLE department ADD INDEX dept_name_idx (name);

Unique indexes

Unique indexes are indexes that help maintain data integrity by ensuring that no two rows of data in a table have identical key values, the exception is that NULL values may occur multiple times.

ALTER TABLE department ADD UNIQUE dept_name_idx (name);

A PRIMARY KEY also is a unique-valued index. It is similar to a UNIQUE index, but has additional restrictions:

A table may have multiple UNIQUE indexes, but at most one PRIMARY KEY.

A UNIQUE index can contain NULL values, whereas a PRIMARY KEY cannot.

A FULLTEXT index is specially designed for text searching.

SHOW INDEX FROM department \G

Index implementation

B-tree indexes

Branch nodes are used for navigating the tree, while leaf nodes hold the actual values and location information.

Bitmap indexes

Although B-tree indexes are great at handling columns that contain many different values, they can become unwieldy when built on a column that allows only a small number of values.

For columns that contain only a small number of values across a large number of rows (known as low-cardinality data), Oracle Database includes bitmap indexes, which generate a bitmap for each value stored in the column.

CREATE BITMAP INDEX acc_prod_idx ON account (product_cd);

Clustered Indexes vs Non-clustered Indexes

http://www.devtoolshed.com/content/clustered-index-vs-non-clustered-index-sql-server

Clustered Index Structures

Nonclustered Index Structures

Clustered Indexes

A clustered index determines the physical storage order of the data in a table.

A table can only have a single clustered index.

The leaf nodes of a clustered index contain the data pages of the underlying table.

By default, primary key creates a clustered index.

Query on clustered index is fast, as it only requires one lookup.

Insert is slower, as the insert must be added in the exact right place in the clustered index.

Non-clustered Index

Non-clustered indexes have the same B-tree structure as clustered indexes, except:

The data rows of the underlying table are not sorted and stored in order based on their non0clustered keys.

A table can contain multiple (up to 255) non-clustered index.

The leaf nodes of non-clustered Index do not consist of the data pages, only contain index pages:

If clustered index present, non-clustered index points back to the data pages in the clustered index.

If no clustered index, non-clustered indexes point to the actual data in the table.

Logical order of the index does not match the physical stored order of the rows on disk.

A unique key by default creates a non-clustered index.

A non-clustered Index has separate storage space different from the table storage.

This means that non-clustered indexes require a lot more hard disk space compared to clustered indexes.

Query on non-clustered indexes requires two lookups. First a lookup for the non-clustered index itself, then a lookup for the primary key.

Advantages and disadvantages of indexes

http://publib.boulder.ibm.com/infocenter/db2luw/v8/index.jsp?topic=/com.ibm.db2.udb.doc/admin/c0005052.htm

http://www.mysqlfaqs.net/mysql-faqs/Indexes/What-are-advantages-and-disadvantages-of-indexes-in-MySQL

The optimizer chooses an index scan if the index columns are referenced in the SELECT statement and if the optimizer estimates that an index scan will be faster than a table scan.

Advantages of Index

Query optimization: Indexes make search queries much faster, because normally Index files generally are smaller and require less time to read than an entire table, particularly as tables grow larger. In addition, the entire index may not need to be scanned. The predicates that are applied to the index reduce the number of rows to be read from the data pages.

Uniqueness: Indexes like primary key index and unique index help to avoid duplicate row data.

Disadvantages of indexes

Every time a row is added, removed or updated, all indexes on that table must be modified. Therefore, the more indexes you have the more work the server needs to do to keep all schema objects up-to-date, which tends to slow things down.

Each index requires storage or disk space. The exact amount depends on the size of the table and the size and number of columns in the index.

Each index potentially adds an alternative access path for a query for the optimizer to consider, which increases the compilation time.

The best strategy is to add an index when a clear need arises. If you need an index for only special purposes, such as a monthly maintenance routine, you can always add the index, run the routine, and then drop the index until you need it again.

Types of Constraints

Constraint is a rule which can not be violated by end users.

There are five types of constraints:

Not null constraints:-which does not allows NULL values.

Unique constraints:-which does not allow duplication but allows NULL values.

Primary key constraints:-the key which does not allow duplication and null values. One table can only have one primary key.

Foreign key constraints:-the key used to refer primary key defined field in another table and it allows duplication.

Check constraint

Restrict the allowable values for a column

Practice

CONSTRAINT fk_product_type_cd FOREIGN KEY (product_type_cd)

REFERENCES product_type (product_type_cd),

CONSTRAINT pk_product PRIMARY KEY (product_cd)

ALTER TABLE product

ADD CONSTRAINT pk_product PRIMARY KEY (product_cd);

ALTER TABLE product

ADD CONSTRAINT fk_product_type_cd FOREIGN KEY (product_type_cd)

REFERENCES product_type (product_type_cd);

Constraints and Indexes

Constraint creation sometimes involves the automatic generation of an index. However, database servers behave differently regarding the relationship between constraints and indexes.

Cascading Constraints

ALTER TABLE product

ADD CONSTRAINT fk_product_type_cd FOREIGN KEY (product_type_cd)

REFERENCES product_type (product_type_cd)

ON UPDATE CASCADE

ON DELETE CASCADE;

View

http://en.wikipedia.org/wiki/View_%28database%29

http://manuals.sybase.com/onlinebooks/group-as/asg1250e/sqlug/@Generic__BookTextView/28457;pt=28420/*

http://wiki.answers.com/Q/What_are_the_advantages_and_disadvantages_of_views_in_a_database

A view consists of a stored query accessible as a virtual table composed of the result set of a query. Unlike ordinary tables (base tables), a view is a virtual table, does not form part of the physical schema: it is a dynamic, virtual table computed or collated from data in the database.

Advantages of View

Views can provide advantages over tables:

* Views can be used to make complex queries easy. A user can use a simple query on a view to display data from multiple tables without having the knowledge of how to join tables in queries.

* Views can simplify Statements for User, we can define frequently used joins, projections, and selections as views so that users do not have to specify all the conditions and qualifications each time an operation is performed on that data.

* Focus on the data that interests them and on the tasks for which they are responsible. Data that is not of interest to a user can be left out of the view.

* Different views can be created from the same data per the requirements, in this way we can display different data for different users.

* Provide additional level of security, through a view, users can query and modify only the data they can see. The rest of the database is neither visible nor accessible, so we can hide sensitive data from certain groups of users.

* Views take very little space to store; the database contains only the definition of a view, not a copy of all the data it presents.

Disadvantages of Views

* View affects performance, querying from view takes more time than directly querying from the table.

* View depends on the table objects, when table is dropped, the view becomes inactive.

* Rows available through a view are not sorted and are not ordered either, we can not use order by when define view.

Read-only vs. updatable views

We can define views as read-only or updatable.

Materialized views

Oracle, DB2 supports materialized views - pre-executed, non-virtual views commonly used in data warehousing.

In materialized view, the query result is cached as a concrete table that may be updated from the original base tables from time to time. This enables much more efficient access, at the cost of some data being potentially out-of-date. The accuracy of a materialized view depends on the frequency or trigger mechanisms behind its updates.

It is most useful in data warehousing scenarios, where frequent queries of the actual base tables can be extremely expensive.

Stored Procedure

http://publib.boulder.ibm.com/infocenter/db2luw/v8/index.jsp?topic=/com.ibm.db2.udb.dc.doc/dc/c_spbenefits.htm

http://searchsqlserver.techtarget.com/news/1052737/Why-use-stored-procedures

http://en.wikipedia.org/wiki/Stored_procedure

A stored procedure is a precompiled subroutine on in database. Stored procedures are used to consolidate and centralize logic that was originally implemented in applications. Large or complex processing that might require the execution of several SQL statements can be moved into stored procedures, and all applications call the procedures only.

Benefits of using stored procedures

Reduced network usage between clients and servers

The stored procedure performs processing on the database server, without transmitting unnecessary data across the network. Only the records that are actually required by the client application are transmitted. Using a stored procedure can result in reduced network usage and better overall performance.

The more SQL statements that you group together in a stored procedure, the more you reduce network usage and the time that database locks are held.

Stored procedures are tunable. By having procedures that handle the database work for your interface, you eliminate the need to modify application source code to improve a query's performance. Changes can be made to the stored procedures that are transparent to the front-end interface.

Stored procedures are usually written by database developers/administrators. Persons holding these roles are usually more experienced in writing efficient queries and SQL statements. If you have your people performing the tasks to which they are best suited, then you will ultimately produce a better overall application.

Improved security, by including database privileges with stored procedures that use static SQL, the database administrator (DBA) can improve security, because you can enable controlled access to sensitive data by appropriate selection of the privileges a program has when it executes.

Centralized security, administration, and maintenance for common routines, by managing shared logic in one place at the server, you can simplify security, administration, and maintenance.

Disadvantages:

We encapsulate business logic in stored procedure, but stored procedures are incredibly tightly coupled to the specific database, it would be hard to switch from one database to another database vendor.

Trigger

http://en.wikipedia.org/wiki/Database_trigger

http://www.allinterview.com/showanswers/20053.html

A database trigger is procedural code that is automatically executed in response to certain events on a particular table or view in a database. Triggers are commonly used to prevent changes, log changes, audit changes, enhance changes, enforce or execute business rules.

Advantages of Triggers

The Main advantage of the trigger is automatic, whenever the table is affected by inserts update, delete, or query that time, the triggers will implicitly call.

Disadvantages of Triggers

Triggers run on every update made to the table therefore it adds more load to the database and cause the system to run slower.

It is not possible to track or debug triggers.

Viewing a trigger is difficult compared to tables, views stored procedures.

Triggers execution is invisible; it is easy to forget about triggers.

Difference between stored-procedure and trigger

They serve two different purposes. A procedure executes only when called. A trigger is 'triggered' by a system event and allows to intercede on insert, update, delete.

We can write a trigger that calls a procedure. We can't really write a procedure that 'calls' a trigger.

Notes on Annotations

2011-06-08T05:59:00.000-07:00

Notes on Annotations

Annotations

Annotations are tags that we insert into we source code so that some tool can process them. The tools can operate on the source level, or they can process class files into which the compiler has placed annotations.

Annotations provide a formalized way to add information to the code so that we can easily use that data at some later point.

Annotations are partly motivated by a general trend toward combining metadata with source-code files, instead of keeping it in external documents.

Using annotations, we can keep this metadata in the Java source code, and have the advantage of cleaner looking code, compile-time type checking and the annotation API to help build processing tools for our annotations.

To benefit from annotations, we need to select a processing tool. We insert annotations into our code that our processing tool understands, and then apply the processing tool.

Annotations can be processed at runtime, at the source code level, or at the bytecode level.

Defining annotations

Defining annotations is very easy; Annotation type declarations are similar to normal interface declarations. An at-sign (@) precedes the interface keyword. Then we declare meta-annotations on the annotation interface such as @Target to define where we can apply this annotation (a method or a field, for example), @Retention to define whether the annotations are available in the source code (SOURCE), in the class files (CLASS), or at run time (RUNTIME).

Then we declare elements in this annotation interface, elements look like interface methods, except that we can declare default values.

Methods can have default values. Method declarations must not have any parameters or a throws clause. Return types are restricted to primitives, String, Class, enums, annotations, and arrays of the preceding types.

Annotations can be read from source files, class files, or reflectively at run time.

The annotations don't do anything by themselves, without tools to read them; annotations are hardly more useful than comments.

We need write annotation processors to read and process annotations. Java SE5 provides extensions to the reflection API to read annotations defined in java class.

Annotation Syntax

An annotation without any elements is called a marker annotation.

An annotation is defined by an annotation interface, Each element declaration has the form: type elementName() default value;

Each annotation has the format

@AnnotationName(elementName1=value1, elementName2=value2, . . .)

The order of the elements does not matter.

Two special shortcuts can simplify annotations:

If no elements are specified, then we don't need to use parentheses: @BugReport

The other shortcut is the single value annotation. If an element has the special name value, and no other element is specified, then we can omit the element name and the = symbol.

@ActionListenerFor("yellowButton") is same as @ActionListenerFor(value="yellowButton")

Limitations when implement annotations

The methods of an annotation interface can have no parameters and no throws clauses, and they cannot be generic.

The type of an annotation element is one of the following: A primitive type, String, Class (with an optional type parameter such as Class), an enum type, an annotation type, 1-dimensional arrays of the preceding types (an array of arrays is not a legal element type)

The compiler will report an error if you try to use any other types. Note that you are not allowed to use any of the wrapper classes, but because of autoboxing this isn’t really a limitation. You can also have elements that are themselves annotations.

//When a Target meta-annotation is not present on an

//annotation type declaration, the declared type may be used on any

//program element.

//If no Retention annotation is present on an annotation type declaration,

//the retention policy defaults to RetentionPolicy.CLASS.

public @interface BugReport {

enum Status {

UNCONFIRMED, CONFIRMED, FIXED, NOTABUG

};

boolean showStopper() default false;

String assignedTo() default "[none]";

Class testCase() default Void.class; // we can use class

Status status() default Status.UNCONFIRMED; // we can use enum

Reference ref() default @Reference(); // an annotation type

String[] reportedBy();

// Compile error: Invalid type Object for the annotation attribute

// BugReport.object; only primitive type, String,

// Class, annotation, enumeration are permitted or 1-dimensional arrays thereof

Object object();

}

Annotation values must be compile-time constants, so you can't call method to set annotation values.

// Compile error: The value for annotation attribute ActionListenerFor.source must be a constant expression

@ActionListenerFor(source = "blueButton" + anyMethod())

An annotation element can never be set to null. Not even a default of null is permissible.

Annotations don’t support inheritance, you cannot use the “extends” keyword with @interfaces.

If annotations (retention class) are evaluated by the compiler, all element values must be compile-time constants.

An item can have multiple annotations, provided they belong to different types. We cannot use the same annotation type more than once when annotating a particular item.

If this is a problem, we can design an annotation that has a value of an array of simpler annotations.

Default value constraints

The compiler is quite picky about default element values. No element can have an unspecified value. This means that elements must either have default values or values provided by the class that uses the annotation.

There is another restriction, which is that none of the non-primitive type elements are allowed to take null as a value, either when declared in the source code or when defined as a default value in the annotation interface. You need define special value as default value.

It is legal to apply an annotation to itself. For example, the @Documented annotation is itself annotated as @Documented.

Annotations for local variables can only be processed at the source level. Class files do not describe local variables. Therefore, all local variable annotations are discarded when a class is compiled. Similarly, annotations for packages are not retained beyond the source level.

Implementation detail

All annotation interfaces implicitly extend the interface java.lang.annotation.Annotation. That interface is a regular interface, not an annotation interface.

Standard Annotations

Meta-Annotations

@Target

The @Target meta-annotation restricts the items to which the annotation applies.

@Target(ElementType.ANNOTATION_TYPE)

public enum ElementType

{TYPE,FIELD,METHOD,PARAMETER,CONSTRUCTOR,LOCAL_VARIABLE,,ANNOTATION_TYPE,PACKAGE}

An annotation without an @Target restriction can be applied to any program element. The compiler checks that we apply an annotation only to a permitted item.

@Retention

The @Retention meta-annotation specifies how long an annotation is retained. We specify at most one of the values.

@Retention(RetentionPolicy.RUNTIME)

SOURCE Annotations are not included in class files.

CLASS Annotations are included in class files, but the virtual machine need not load them.

RUNTIME Annotations are included in class files and loaded by the virtual machine. They are available through the reflection API.

public enum RetentionPolicy {

* Annotations are to be discarded by the compiler.

SOURCE,

* Annotations are to be recorded in the class file by the compiler

* but need not be retained by the VM at run time. This is the default behavior.

CLASS,

* Annotations are to be recorded in the class file by the compiler and

* retained by the VM at run time, so they may be read reflectively.

RUNTIME

}

If no Retention annotation is present on an annotation type declaration, the retention policy defaults to RetentionPolicy.CLASS.

The @Documented meta-annotation gives a hint to documentation tools such as Javadoc.

@Inherited

The @Inherited meta-annotation applies only to annotations for classes. When a class has an inherited annotation, then all of its subclasses automatically have the same annotation.

Annotations for Compilation

@Deprecated

The @SuppressWarnings annotation tells the compiler to suppress warnings of a particular type: @SuppressWarnings("unchecked")

The @Override annotation applies only to methods. The compiler checks that a method with this annotation really overrides a method from the superclass.

The @Generated annotation is intended for use by code generator tools. Any generated source code can be annotated to differentiate it from programmer-provided code.

Annotations for Managing Resources

@PostConstruct, @PreDestroy, @Resource

Practical usage

Source-Level Annotation Processing

One use for annotation is the automatic generation of "side files" that contain additional information about programs

Bytecode Engineering

We can process class files using Bytecode Engineering Library (BCEL).

Generating external files

Annotation based command line interface parser

How Junit process annotation.

Resources:

Core Java, Vol. 2: Advanced Features, 8th Edition

Thinking in Java

Notes on Hadoop

2011-06-06T10:02:00.001-07:00

Notes on Hadoop

What is Hadoop?

Hadoop is an open source framework for writing and running distributed applications that process large amounts of data.

Hadoop is designed for data-intensive processing, and it adopts move-code-to-data philosophy.

Its main features are distributed storage and distributed computing.

1. Accessible, hadoop runs on large clusters of (low-end) commodity machines.

2. It's robust(Fault Tolerance). It can gracefully handle frequent hardware malfunction and failures.

3. It's scalable(Scalability), Hadoop scales linearly to handle larger data by adding more nodes to the cluster.

4. Simple—Hadoop allows users to quickly write efficient parallel code.

By using distributed storage and transferring code instead of data, Hadoop avoids the costly transmission step when working with large data sets. Moreover, the redundancy of data allows Hadoop to recover should a single node fail.

It's east to create programs with Hadoop MapReduce framework.

You don't have to worry about partitioning the data, determining which nodes will perform which tasks, or handling communication between nodes. Hadoop handles this for you, leaving you free to focus on what’s most important to you—your data and what you want to do with it.

Hadoop is a core part of the computing infrastructure for many web companies, such as Yahoo, Facebook, LinkedIn, and Twitte.

Compared with Parallel Databases

A few scales to low hundreds of nodes and about 5 PB

Primary design goal is “performance”

Requires homogeneous hardware

Anomalous behavior is not well tolerated:

  A slow network can cause serious performance degradation

  Most queries fail when one node fails

What is MapReduce?

MapReduce is a programming model and software framework for writing applications that rapidly process vast amounts of data in parallel on large clusters of compute nodes.

HDFS

Hadoop Distributed File System is the primary storage system used by Hadoop applications.

HDFS divides data set into smaller (typically 64 MB) blocks that are spread among many machines in the cluster.

HDFS creates multiple replicas of data blocks and distributes them on compute nodes throughout a cluster.

This way, the data can be read in parallel and it provides a much higher throughput, also it is much more reliable, as even when one node fails, it can still read the data's replica from other nodes.

And using HDFS is also much cheaper, as a cluster of commodity machines turns out to be much cheaper than one high-end server!

Computationally intensive processing vs data-intensive processing

Hadoop is designed for data-intensive processing, and it adopts move-code-to-data philosophy.

The programs to run (“code”) are much smaller than the data and are easier to move around.

Let the data remain where it is and move the executable code to its hosting machine.

The clients send only the MapReduce programs to be executed, and these programs are usually small (often in kilobytes).

Data is broken up and distributed across the cluster, and as much as possible, computation on a piece of data takes place on the same machine where that piece of data resides.

SCALE-OUT INSTEAD OF SCALE-UP

Hadoop is designed to be a scale-out architecture operating on a cluster of com-

modity PC machines. Adding more resources means adding more machines to the Hadoop cluster. Hadoop clusters with ten to hundreds of machines is standard.

KEY/VALUE PAIRS INSTEAD OF RELATIONAL TABLES

Hadoop uses key/value pairs as its basic data unit, which is flexible enough to work with the less-structured data types.

FUNCTIONAL PROGRAMMING (MAPREDUCE) INSTEAD OF DECLARATIVE QUERIES (SQL)

Hadoop use MapReduce, Under MapReduce you specify the actual steps in processing the data.

Under SQL you have query statements; under MapReduce you have scripts and codes.

OFFLINE BATCH PROCESSING INSTEAD OF ONLINE TRANSACTIONS

Hadoop is designed for offline processing and analysis of large-scale data. It doesn’t work for random reading and writing of a few records, which is the type of load for online transaction processing.

Hadoop is best used as a write-once, read-many-times type of data store.

SQL database is by design targeted at structured data. Many of Hadoop’s initial applications deal with unstructured data such as text.

Understanding MapReduce

Mapper, reducer, combiner, patitioner

MapReduce programs are executed in two main phases, called mapping and reducing. Each phase is defined by a data processing function, and these functions are called mapper and reducer, respectively. In the mapping phase, MapReduce takes the input data and feeds each data element to the mapper. In the reducing phase, the reducer processes all the outputs from the mapper and arrives at a final result.

In simple terms, the mapper is meant to filter and transform the input into something that the reducer can aggregate over.

The building blocks of Hadoop

NameNode

Hadoop employs a master/slave architecture, The NameNode is the master of HDFS that directs the slave DataNode daemons to perform the low-level I/O tasks. The NameNode is the bookkeeper of HDFS; it keeps track of how your files are broken down into file blocks, which nodes store those blocks, and the overall health of the distributed filesystem.

It doesn’t store any user data or perform any computations for a MapReduce program to lower the workload on the machine.

The NameNode keeps track of the file metadata—which files are in the system and how each file is broken down into blocks.

NameNode is a single point of failure of your Hadoop cluster.

DataNode

In HDFS file system, the file is broken into blocks and distributed among the DataNodes.

The NameNode will tell your client which DataNode each block resides in.

Your client communicates directly with the DataNode daemons to process the local files corresponding to the blocks. Furthermore, a DataNode may communicate with other DataNodes to replicate its data blocks for redundancy.

The DataNodes provide backup store of the blocks and constantly report to the NameNode to keep the metadata current.

Upon initialization, each of the DataNodes informs the NameNode of the blocks it’s currently storing. After this mapping is complete, the DataNodes continually poll the NameNode to provide information regarding local changes as well as receive instructions to create, move, or delete blocks from the local disk.

Secondary NameNode - SNN

The Secondary NameNode (SNN) is an assistant daemon for monitoring the state of the cluster HDFS.

The SNN differs from the NameNode in that this process doesn’t receive or record any real-time changes to HDFS. Instead, it communicates with the NameNode to take snapshots of the HDFS metadata at intervals defined by the cluster configuration.

The NameNode is a single point of failure for a Hadoop cluster, and the SNN snapshots help minimize the downtime and loss of data. Nevertheless, a NameNode failure requires human intervention to reconfigure the cluster to use the SNN as the primary NameNode

JobTracker

The JobTracker daemon is the liaison between your application and Hadoop. Once you submit your code to your cluster, the JobTracker determines the execution plan by determining which files to process, assigns nodes to different tasks, and monitors all tasks as they’re running. Should a task fail, the JobTracker will automatically relaunch the task, possibly on a different node, up to a predefined limit of retries.

There is only one JobTracker daemon per Hadoop cluster. It’s typically run on a server as a master node of the cluster.

TaskTracker

The computing daemons also follow a master/slave architecture: the JobTracker is the master overseeing the overall execution of a MapReduce job and the TaskTrackers manage the execution of individual tasks on each slave node.

Each TaskTracker is responsible for executing the individual tasks that the JobTracker assigns. Although there is a single TaskTracker per slave node, each TaskTracker can spawn multiple JVMs to handle many map or reduce tasks in parallel.

One responsibility of the TaskTracker is to constantly communicate with the

JobTracker. If the JobTracker fails to receive a heartbeat from a TaskTracker within a specified amount of time, it will assume the TaskTracker has crashed and will resubmit the corresponding tasks to other nodes in the cluster.

NameNode: Maps a file to a file-id and list of DataNodes

DataNode: Maps a block-id to a physical location on disk

SecondaryNameNode: Periodic merge of Transaction log

Distributed File System

  Single Namespace for entire cluster

  Data Coherency

– Write-once-read-many access model

– Client can only append to existing files

  Files are broken up into blocks

– Typically 128 MB block size

– Each block replicated on multiple DataNodes

  Intelligent Client

– Client can find location of blocks

– Client accesses data directly from DataNode

NameNode Metadata

  Meta-data in Memory

– The entire metadata is in main memory

– No demand paging of meta-data

  Types of Metadata

– List of files

– List of Blocks for each file

– List of DataNodes for each block

– File attributes, e.g creation time, replication factor

  A Transaction Log

– Records file creations, file deletions. etc

DataNode

  A Block Server

– Stores data in the local file system (e.g. ext3)

– Stores meta-data of a block (e.g. CRC32)

– Serves data and meta-data to Clients

- Periodic validation of checksums

  Block Report

– Periodically sends a report of all existing blocks to t

NameNode

  Facilitates Pipelining of Data

– Forwards data to other specified DataNodes

Block Placement

  Current Strategy

-- One replica on local node

-- Second replica on a remote rack

-- Third replica on same remote rack

-- Additional replicas are randomly placed

  Clients read from nearest replica

  Pluggable policy for placing block replicas

Data Pipelining

  Client writes block to the first DataNode

  The first DataNode forwards the data to the next DataNode in the Pipeline, and so on

  When all replicas are written, the Client moves on to write the next block in file

Topology

This topology features a master node running the NameNode and JobTracker daemons and a standalone node with the SNN in case the master node fails.

Hadoop can be run in standalone mode, Pseudo-distributed mode, Fully distributed mode.

Local (standalone) mode

The standalone mode is the default mode for Hadoop.

Because there’s no need to communicate with other nodes, the standalone mode doesn’t use HDFS, nor will it launch any of the Hadoop daemons. Its primary use is for developing and debugging the application logic of a MapReduce program without the additional complexity of interacting with the daemons.

Pseudo-distributed mode

The pseudo-distributed mode is running Hadoop in a “cluster of one” with all daemons running on a single machine.

This mode complements the standalone mode for debugging your code, allowing you to examine memory usage, HDFS input/output issues, and other daemon interactions.

Setup Hadoop cluster

Master node communicates with other nodes via passphraseless SSH.

The public key is stored locally on every node in the cluster, and the master node sends the private key when attempting to access a remote machine.

Define a common account on all nodes

Generate SSH key pair, distribute public key to other nodes, and validate logins.

scp ~/.ssh/id_rsa.pub hadoop-user@target:~/master_key

mv ~/master_key ~/.ssh/authorized_keys

chmod 600 ~/.ssh/authorized_keys

Running Hadoop

Switching between modes

Create a separate configuration folder for each of the modes and place the appropriate version of the XML files in the corresponding folder, then create symbolic links to switch between Hadoop modes

Web-based cluster UI

http://localhost:50070/

Working with files in HDFS

HDFS is a filesystem designed for large-scale distributed data processing under frameworks such as MapReduce. You can store a big data set of (say) 100 TB as a single file in HDFS.

As HDFS isn’t a native Unix filesystem, standard Unix file tools don’t work on it,

Hadoop does provide a set of command line utilities.

There is a project MountableHDFS that is trying to make HDFS mountable as a Unix filesystem.

Hadoop commands

Hadoop file commands can interact with both the HDFS filesystem and the local

filesystem.

Edit the script conf/hadoop-env.sh to set JAVA_HOME

Run in standlone mode

hadoop jar hadoop-examples-*.jar wordcount /home/essniuser/dscli/log output

hadoop namenode -format

jps,start-all.sh

hadoop fs -ls, -lsr

hadoop fs -cat hdfs://localhost:9000/user/chuck/example.txt

hadoop fs -mkdir /user/chuck, -put example.txt., -get example.txt.-cat example.txt, -cat example.txt | head, -tail example.txt, –rm example.txt

In general, Hadoop works more effectively with a single large file rather than a number of smaller ones.

Configuration conf = new Configuration(); // Factory method pattern

FileSystem hdfs = FileSystem.get(conf); // get HDFS fiel system

FileSystem local = FileSystem.getLocal(conf); // get local file system

Anatomy of a MapReduce program

a MapReduce program processes data by manipulating

(key/value) pairs in the general form

map: (K1,V1) -> list(K2,V2)

reduce: (K2,list(V2)) -> list(K3,V3)

After distributing input data to different nodes, the only time nodes communicate with each other is at the “shuffle” step. This restriction on communication greatly helps scalability.

Hadoop data types . The MapReduce framework won’t allow keys or values to be any arbitrary class.

This is because the MapReduce framework has a certain defined way of serializing the key/value pairs to move them across the cluster’s network, and only classes that support this kind of serialization can function as keys or values in the framework.

More specifically, classes that implement the Writable interface can be values, and classes that implement the WritableComparable interface can be either keys or values.

Predefined data types

BooleanWritable, ByteWritable, DoubleWritable, FloatWritable, IntWritable, LongWritable, Text, NullWritable

You can create your own custom type as long as it implements the Writable

(or WritableComparable) interface.

Mapper

To serve as the mapper, a class implements from the Mapper interface and inherits the MapReduceBase class

In Mapper, we need implement its map function, which generates a (possibly empty) list of (K2, V2) pairs for a given (K1, V1) input pair.

Predefined Mappper

IdentityMapper, InverseMapper, RegexMapper, TokenCountMapper

Reducer

void reduce(K2 key,Iterator values,OutputCollector output, Reporter reporter) throws IOException

When the reducer task receives the output from the various mappers, it sorts the incoming data on the key of the (key/value) pair and groups together all values of the same key. The reduce() function is then called, and it generates a (possibly empty) list of (K3, V3) pairs by iterating over the values associated with a given key.

IdentityReducer, LongSumReducer

Partitioner— redirecting output from Mapper

A common misconception for first-time MapReduce programmers is to use only a single reducer.

With multiple reducers, we need some way to determine the appropriate one to send a (key/value) pair outputted by a mapper. The default behavior is to hash the key to determine the reducer. Hadoop enforces this strategy by use of the HashPartitioner class, we can define our own Partitioner.

It defines method that returns an integer between 0 and the number of reduce tasks indexing to which reducer the (key/value) pair will be sent.

Shuffling

Between the map and reduce stages, a MapReduce application must take the output from the mapper tasks and distribute the results among the reducer tasks. This process is typically called shuffling, because the output of a mapper on a single node may be sent to reducers across multiple nodes in the cluster.

Combiner—local reduce

Perform a “local reduce“ before we distribute the mapper results, this can reduce the results/data that are distributed to reducers.

One of the fundamental principles of MapReduce’s processing power is the splitting of the input data into chunks. You can process these chunks in parallel using multiple machines.

HDFS provides efficient way to random read access.

HDFS stores files in blocks spread over multiple machines. Roughly speaking, each file block is a split. As different machines will likely have different blocks, parallelization is automatic if each split/block is processed by the machine that it’s residing at. Furthermore, as HDFS replicates blocks in multiple nodes for reliability, MapReduce can choose any of the nodes that have a copy of a split/block.

InputFormat

TextInputFormat: Key is the byte offset of the line, and value is the content of the line.

KeyValueTextInputFormat, SequenceFileInputFormat

NLineInputFormat: Same as TextInputFormat, but each split is guaranteed to have exactly N lines.

OutputFormat

The output has no splits, as each reducer writes its output only to its own file. The output files reside in a common directory and are typically named part-nnnnn, where nnnnn is the partition ID of the reducer.

RecordWriter

Objects format the output and RecordReaders parse the format of the input.

TextOutputFormat, SequenceFileOutputFormat, NullOutputFormat

Writing basic MapReduce programs

In general, MapReduce programs follows basic template, we generally take an existing MapReduce program and modify it until it does what we want.

Adapting for Hadoop’s API changes[How]

The 0.20 release supports the future

API while maintaining backward-compatibility with the old one by marking it as deprecated.

Streaming in Hadoop

Hadoop supports other languages via a generic API called Streaming.

Hadoop Streaming interacts with programs using the Unix streaming paradigm.

Inputs come in through STDIN and outputs go to STDOUT.

Streaming with Unix commands

Under Streaming, each mapper sees the entire stream of data, and it’s the mapper that takes on the responsibility of breaking the stream into (line-oriented) records.

In the standard Java model, the framework itself breaks input data into records, and gives the map() method only one record at a time. The Streaming model makes it easy to keep state information across records in a split.

hadoop jar contrib/streaming/hadoop-0.19.1-streaming.jar -input input/cite75_99.txt -output output -mapper 'cut -f 2 -d,' -reducer 'uniq'

hadoop jar hadoop-0.19.1-streaming.jar -input output -output output_a -mapper 'wc -l' -D mapred.reduce.tasks=0

We want the mapper to directly output the record count without any reducer, so we set mapred.reduce.tasks to 0 and don’t specify the -reducer option at all.

Streaming with scripts

hadoop jar hadoop-0.19.1-streaming.jar -input input/cite75_99.txt -output output -mapper 'RandomSample.py 10' -file RandomSample.py -D mapred.reduce.tasks=1

hadoop jar hadoop-0.18.1-streaming.jar -input input/apat63_99.txt -output output -mapper 'AttributeMax.py 8' -reducer 'AttributeMax.py 0' -file AttributeMax.py -D mapred.reduce.tasks=1

As we haven’t specified any particular reducer, it will use the default IdentityReducer.

Classes of aggregation functions: distributive, algebraic, and holistic

Streaming with the Aggregate package

You only have to provide a mapper that processes records and sends out a specially formatted output. Each line of the mapper’s output looks like

function:key\tvalue.

Improving performance with combiners

It’s supposed to whittle down the output of the mapper to lessen the load on the network and on the reducer.

If we specify a combiner, the MapReduce framework may apply it to the intermediate data from mapper.

Chaining Multiple MapReduce jobs

Chain multiple MapReduce jobs to run sequentially, with the output of one MapReduce job being the input to the next.

As JobClient.runJob() blocks until the end of a job, chaining MapReduce jobs involves calling the driver of one MapReduce job after another. The driver at each job will have to create a new JobConf object and set its input path to be the output path of the previous job. You can delete the intermediate data generated at each step of the chain at the end.

Chaining MapReduce jobs with complex dependency

In addition to holding job configuration information, Job also holds dependency information, specified through the addDependingJob() method.

JobControl objects do the managing and monitoring of the job execution. You can add jobs to a JobControl object via the addJob() method.

After adding all the jobs and dependencies, call JobControl’s run() method to spawn a thread to submit and monitor jobs for execution. JobControl has methods like allFinished() and getFailedJobs() to track the execution of various jobs within the batch.

Joining data from different sources

Reduce-side joining

Hadoop has a contrib package called datajoin that works as a generic framework for data joining in Hadoop. Its jar file is at contrib/datajoin/hadoop-*-datajoin.jar.

Hadoop MapReduce code example

public class CitationHistogram extends Configured implements Tool {

public static class MapClass extends MapReduceBase

implements Mapper {

private final static IntWritable uno = new IntWritable(1);

private IntWritable citationCount = new IntWritable();

// these two are defined to improve preformance

// The map() method will be called as many times as there are records

// (in a split, for each JVM). Reducing the number of objects created

// inside the map() method can increase performance and reduce

// garbage collection .

public void map(Text key, Text value,

OutputCollector output,

Reporter reporter) throws IOException {

citationCount.set(Integer.parseInt(value.toString()));

output.collect(citationCount, uno);

}

public static class Reduce extends MapReduceBase

implements Reducer

{

public void reduce(IntWritable key, Iterator values,

OutputCollectoroutput,

Reporter reporter) throws IOException {

int count = 0;

while (values.hasNext()) {

count += values.next().get();

}

output.collect(key, new IntWritable(count));

}

public int run(String[] args) throws Exception {

Configuration conf = getConf();

JobConf job = new JobConf(conf, CitationHistogram.class);

Path in = new Path(args[0]);

Path out = new Path(args[1]);

FileInputFormat.setInputPaths(job, in);

FileOutputFormat.setOutputPath(job, out);

job.setJobName("CitationHistogram");

job.setMapperClass(MapClass.class);

job.setReducerClass(Reduce.class);

job.setInputFormat(KeyValueTextInputFormat.class);

job.setOutputFormat(TextOutputFormat.class);

job.setOutputKeyClass(IntWritable.class);

job.setOutputValueClass(IntWritable.class);

JobClient.runJob(job);

return 0;

}

public static void main(String[] args) throws Exception {

int res = ToolRunner.run(new Configuration(),

new CitationHistogram(),

args);

System.exit(res);

}

Revised version of the WordCount example

public class WordCount2 {

public static void main(String[] args) {

JobClient client = new JobClient();

JobConf conf = new JobConf(WordCount2.class);

FileInputFormat.addInputPath(conf, new Path(args[0]));

FileOutputFormat.setOutputPath(conf, new Path(args[1]));

conf.setOutputKeyClass(Text.class);

conf.setOutputValueClass(LongWritable.class);

conf.setMapperClass(TokenCountMapper.class); q conf.setCombinerClass(LongSumReducer.class);

conf.setReducerClass(LongSumReducer.class); w

client.setConf(conf);

try {

JobClient.runJob(conf);

} catch (Exception e) {

e.printStackTrace();

}

Resource

Hadoop in Action

Notes on Memcached

2011-06-03T20:36:00.000-07:00

Notes on Memcached

What is Memcached?

memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

How Memcached works?

Memcached's magic lies in its two-stage hash approach. It behaves as though it were a giant hash table, looking up key/value pairs. Give it a key, and set or get some arbitrary data.

Memcached is organized as a farm of N servers. The storage model can be considered as a huge HashTable partitioned among these N servers.

The servers within the farm doesn't know each other at all, and it doesn't replicate data between different server, so its overhead is slow, and add, remove a memcached server is very easy.

When doing a memcached lookup, first the client hashes the key against the whole list of servers and choses a server, then the client sends this request, and the server does an internal hash key lookup for the actual item data, and return response.

If the response is not null, we get the data we need, if not, we need query database and get the data and add the data to memcached server.

Each server use asynchronous, non-blocking I/O and a thread pool to handle large number of incoming TCP sockets, the number of threads is independent of the number of incoming sockets.

maxiumum key length - 250 bytes

maxiumum value size - 1 MB

limits on expire time - 30 days

Slabs, Pages, Chunks and Memcached Memory Management

Memcached use slab allocator to do all the memory-management for item data in memcached. It still uses malloc/calloc/realloc for other memory management, just use slab allocator for its key/value pair.

When you ask memcached to store a value, it looks up the “slab” associated with that value. A slab holds values within a particular size range. Slabs are composed of 1MB pages, which are broken into chunks of the slab’s size.

In a simple word, a slab has many pages which has many chunks. Each chunk is a fixed size.

When we insert a value, memcached will look up all slabs to find the slab whose chunk size is larger than and closest to the value size, then will insert the value to that slab's chunk.

Memcached Protocol

1. Text based protocol

The original memcached protocol is a very simple, small, and an effective text based protocol

2. Binary Protocol

Binary protocol is added in memcached 1.3 to improve performance and extensibility.

Parsing binary string is more efficient than parsing text string, and its extensibility opens up new possibilities,

Memcached SASL

What is SASL?

SASL stands for Simple Authentication and Security Layer, which provides you a means of adding authentication support to connection-based protocols, such as the memcached client binary protocol in this case.

memcached server won't allow the user to connect unless they provide correct credential.

Install prerequisites such as sasl development libraries and as well as SASL utilities such as saslpasswd.

sudo apt-get -f install libsasl2-2 sasl2-bin libsasl2-2 libsasl2-dev libsasl2-modules

Compile and install memcached with --enable-sasl

./configure --enable-sasl

Configuring SASL

# Create a user for memcached.

saslpasswd2 -a memcached -c cacheuser

Running Memcached

In order to enable SASL support in the server you must use the -S flag.

The -S flag does a few things things:

Enable all of the SASL commands.

Require binary protocol only.

Require authentication to have been successful before commands may be issued on a connection.

What if memcached server is added, or removed.

The default server hashing algorithm doesn't handle the growth and shrink of the number of servers very well.

It computes hash value of the key and them module it by server number.

When the number of server changes, the ownership equation (key mod N) will all be wrong, as a same key now would be mapped and redistributed to different server.

There are several approaches to this problem:

1. Consistent hashing

Consistent hashing localize the ownership changes to just the neighbor of the added or removed server.

Under this schema, each server is assigned with an id under the same key space. The ownership of a key is determined by the closest server whose key is the first one encountered when walking in the anti-clockwise direction.

When a server crashes, its immediate upstream neighbor server (walking along the anti-clockwise direction) will adopt the key ownership of the dead server, while all other servers have the same ownership of key range unchanged.

When a new server is added, only its neighbor servers’ ownership would change.

2. Run logical servers within a farm of physical machines.

When a physical machine crashes, its logical servers will be restart in the surviving physical machines. In other words, the number of logical servers is unchanged even when crashes happens.

This logical server approach is also good when the underlying physical machines have different memory capacity. We can start more Memcached process in the machine with more memory and proportionally spread the cache according to memory capacity.

What is memcached's cache?

The cache structure is an LRU (Least Recently Used), plus expiration timeouts. When you store items into memcached, you may state how long it should be valid in the cache. Which is forever, or some time in the future. If the server is out of memory, expired slabs are replaced first, then the oldest unused slabs go next.

API

get(key)

set(key, value)

set(key, value, expiration)

add(key, value, expiration)

replace(key, value, expiration)

memcached commands

Start server: memcached -d -m 4096 -t 4 -p 11211

-d[deamon], -t[thread number], -m[memory size] -p[port]

Stop server: ps -ef|grep memcached|grep -v grep|awk '{print $2}'|xargs kill -9

telnet localhost 11211

Query the current traffic statistics: stats

Query the current memory statistics: stats slabs

Query which keys are used: stats items

Command Description Example

get Reads a value get mykey

set Set a key unconditionally set mykey 0 60 5

add Add a new key add newkey 0 60 5

replace Overwrite existing key replace key 0 60 5

append Append data to existing key append key 0 60 15

prepend Prepend data to existing key prepend key 0 60 15

incr Increments numerical key value by given number

incr mykey 2

decr Decrements numerical key value by given number

decr mykey 5

delete Deletes an existing key delete mykey

flush_all Invalidate specific items immediately flush_all

Invalidate all items in n seconds flush_all 900

stats Prints general statistics stats

Prints memory statistics stats slabs

Prints memory statistics stats malloc

stats items,stats detail,stats sizes

Resets statistics stats reset

version Prints server version. version

verbosity Increases log level verbosity

quit Terminate telnet session quit

Install memcached

1. Install Libevent

libevent is an asynchronous event notification software library.

The libevent API provides a mechanism to execute a callback function when a specific event occurs on a file descriptor or after a timeout has been reached. Furthermore, libevent also supports callbacks due to signals or regular timeouts.

wget http://monkey.org/~provos/libevent-2.0.11-stable.tar.gz

cd ibevent-2.0.11-stable

autoconf

./configure --prefix=/usr/local

make

sudo make install

2. Install Memcached

Download the latest version of Memcached:

wget http://memcached.googlecode.com/files/memcached-1.4.5.tar.gz

cd memcached-1.4.5

autoconf

./configure --prefix=/usr/local

make

sudo make install

Using Memcached with Java

There are several memcached java client implementations, such as spymemcached, and Memcached-Java-Client.

Source code Digestion

// Implements Spring InitializingBean and DisposableBean

public class SpyMemcachedClient implements InitializingBean, DisposableBean {

private MemcachedClient memcachedClient;

private String memcachedNodes = "9.186.1.1:11211,9.186.1.2:11211,9.186.1.3:11211";

private boolean isBinaryProtocol = false;

private boolean isConsistentHashing = true;

private long operationTimeout = 1000; //default value in Spy is 1000ms

@Override

public void afterPropertiesSet() throws Exception {

ConnectionFactoryBuilder cfb = new ConnectionFactoryBuilder();

cfb.setFailureMode(FailureMode.Redistribute);

cfb.setDaemon(true);

cfb.setProtocol(isBinaryProtocol ? Protocol.BINARY : Protocol.TEXT);

if (isConsistentHashing) {

cfb.setLocatorType(Locator.CONSISTENT);

cfb.setHashAlg(HashAlgorithm.KETAMA_HASH);

}

cfb.setOpTimeout(operationTimeout);

try {

memcachedClient = new MemcachedClient(cfb.build(), AddrUtil.getAddresses(memcachedNodes));

} catch (IOException e) {

logger.error("MemcachedClient initilization error: ", e);

throw e;

}

public void destroy() throws Exception {

if (memcachedClient != null) {

memcachedClient.shutdown();

}

public MemcachedClient getMemcachedClient() {

return memcachedClient;

}

// capture cast exception, and return null instead

@SuppressWarnings("unchecked")

public T get(String key) {

try {

return (T) memcachedClient.get(key);

} catch (RuntimeException e) {

logger.warn("Get from memcached server fail,key is " + key, e);

return null;

}

// getBulk,gets, set, cas(check and set),delete, incr, decr, asyncIncr, asyncDecr

// setMemcachedNodes, setBinaryProtocol, setConsistentHashing, setOperationTimeout

}

@Component

public class AccountManager {

private SpyMemcachedClient spyMemcachedClient;

@Transactional(readOnly = true)

public User getInitedUser(String id) {

if (spyMemcachedClient != null) {

return getUserFromMemcached(id);

} else {

return userJdbcDao.queryObject(id);

}

private User getUserFromMemcached(String id) {

String key = MemcachedObjectType.USER.getPrefix() + id;

User user = null;

String jsonString = spyMemcachedClient.get(key);

if (jsonString == null) {

user = userJdbcDao.queryObject(id);

if (user != null) {

jsonString = jsonBinder.toJson(user);

spyMemcachedClient.set(key, MemcachedObjectType.USER.getExpiredTime(), jsonString);

}

} else {

user = jsonBinder.fromJson(jsonString, User.class);

}

return user;

}

@Autowired(required = false) //Autowired

public void setSpyMemcachedClient(SpyMemcachedClient spyMemcachedClient) {

this.spyMemcachedClient = spyMemcachedClient;

}

Resources

Memcached全面剖析（Mixi） – 索引

memcached FAQ

Notes on Memcached

Slabs, Pages, Chunks and Memcached

memcached Telnet Interface

Learning JDK Code - Collection

2011-06-02T07:58:00.003-07:00

Learning JDK Code - Collection

HashMap vs Hashtable

http://stackoverflow.com/questions/2414117/explanation-of-hashmaphashint-method

http://apurvagnihotri.blogspot.com/2010/06/hashmap-vs-hashtable.html

The Hashtable is among the original collection classes in Java.Hashtable extends the Dictionary class, which as the Javadocs state, is obsolete and has been replaced by the Map interface. HashMap is part of the new Collections Framework, added with Java 2.

The key difference between the two is that access to the Hashtable is synchronized on the table while access to the HashMap is not synchronized.This makes HashMap better for non-threaded applications, as unsynchronized Objects typically perform better than synchronized ones

HashMap has a more complex hashing algorithm then Hashtable. It takes the hash value from the key and then hashes it again (double hashing). This can improve the distribution of the keys and hence the performance of the Map.

Another difference is that iterator in the HashMap is fail-safe while the enumerator for the Hashtable isn't. If we change the map while iterating, it will throw exception.

Third difference is that HashMap permits null values in it, while Hashtable doesn't. Also note that only one NULL value is allowed as a key in HashMap. HashMap does not allow multiple keys to be NULL. Nevertheless, it can have multiple NULL values.

1. Initial size and resize

HashMap uses power-of-two length tables, default capacity is 16.

static final int DEFAULT_INITIAL_CAPACITY = 16;

static final int MAXIMUM_CAPACITY = 1 << 30;

static final float DEFAULT_LOAD_FACTOR = 0.75f;

public HashMap(int initialCapacity, float loadFactor) {

// Find a power of 2 >= initialCapacity

int capacity = 1;

while (capacity < initialCapacity)

capacity <<= 1;

}

User can specify map size, hashmap would find a pow of 2 whose value is just larger than user specified size.

resize(2 * table.length);

Hashset would construct a new, empty hashtable with the specified initial capacity.

Default size is 11.

public Hashtable(int initialCapacity, float loadFactor) {

if (initialCapacity==0)

initialCapacity = 1;

this.loadFactor = loadFactor;

table = new Entry[initialCapacity];

threshold = (int)(initialCapacity * loadFactor);

}

public Hashtable() {

this(11, 0.75f);

}

int newCapacity = oldCapacity * 2 + 1;

HashMap hash function

The way the HashMap is implemented relies on the hashCode function being sufficiently well implemented. If you have many collisions on the lower bits, the HashMap will not perform well.

Because the implementation of hashCode is outside of the control of HashMap (every object can implement their own), they supply an additional hash function that shifts the object's hashCode around a little to ensure that the lower bits are distributed more randomly.

HashMap uses power-of-two length tables, and hashes keys by masking away the higher bits and taking only the lower bits of their hash code.

So hash() attempts to bring relevancy to the higher bits, which otherwise would get masked away (indexFor basically discards the higher bits of h and takes only the lower k bits where length == (1 << k)).

int hash = hash(key.hashCode());

int i = indexFor(hash, table.length);

* Applies a supplemental hash function to a given hashCode, which

* defends against poor quality hash functions. This is critical

* because HashMap uses power-of-two length hash tables, that

* otherwise encounter collisions for hashCodes that do not differ

* in lower bits. Note: Null keys always map to hash 0, thus index 0.

static int hash(int h) {

// This function ensures that hashCodes that differ only by

// constant multiples at each bit position have a bounded

// number of collisions (approximately 8 at default load factor).

h ^= (h >>> 20) ^ (h >>> 12);

return h ^ (h >>> 7) ^ (h >>> 4);

}

static int indexFor(int h, int length) {

return h & (length-1);

}

Hashtable hash function

By doing the more expensive % operation (instead of simple bit masking), the performance of Hashtable is less sensitive to hash codes with poor distribution in the lower bits (especially if table.length is a prime number).

int hash = key.hashCode();

int index = (hash & 0x7FFFFFFF) % tab.length;

HashMap[Sun JDK]

 public class HashMap extends AbstractMap implements Map, Cloneable, Serializable  
 {  
   // The next size value at which to resize (capacity * load factor).  
   int threshold;    
   public HashMap() {  
     this.loadFactor = DEFAULT_LOAD_FACTOR;  
     threshold = (int)(DEFAULT_INITIAL_CAPACITY * DEFAULT_LOAD_FACTOR); //  
     table = new Entry[DEFAULT_INITIAL_CAPACITY];  
     init(); // subclass would overwite this  
   }  
   public V put(K key, V value) {  
     if (key == null)  
       return putForNullKey(value); // We can put null key and null value to HashMap.  
     int hash = hash(key.hashCode());  
     int i = indexFor(hash, table.length);  
     for (Entry e = table[i]; e != null; e = e.next) {  
       Object k;  
       if (e.hash == hash && ((k = e.key) == key || key.equals(k))) // call e.key == key  
       {  
         V oldValue = e.value;  
         e.value = value;  
         e.recordAccess(this);  
         return oldValue;  
       }  
     }  
     modCount++;  
     addEntry(hash, key, value, i);  
     return null;  
   }  
   public V get(Object key) {  
     if (key == null)  
       return getForNullKey();  
     int hash = hash(key.hashCode());  
     for (Entry e = table[indexFor(hash, table.length)];  
        e != null;  
        e = e.next) {  
       Object k;  
       if (e.hash == hash && ((k = e.key) == key || key.equals(k)))  
         return e.value;  
     }  
     return null;  
   }  
   void addEntry(int hash, K key, V value, int bucketIndex) {  
    Entry e = table[bucketIndex];  
     table[bucketIndex] = new Entry(hash, key, value, e);  
     if (size++ >= threshold)  
       resize(2 * table.length);  
 }  
   public boolean containsKey(Object key) {  
     return getEntry(key) != null;  
   }  
   final Entry<K,V> getEntry(Object key) {  
     int hash = (key == null) ? 0 : hash(key.hashCode());  
     for (Entry<K,V> e = table[indexFor(hash, table.length)];  
        e != null;  
        e = e.next) {  
       Object k;  
       if (e.hash == hash &&  
         ((k = e.key) == key || (key != null && key.equals(k))))  
         return e;  
     }  
     return null;  
 }  
    /**  
    * Applies a supplemental hash function to a given hashCode, which  
    * defends against poor quality hash functions. This is critical  
    * because HashMap uses power-of-two length hash tables, that  
    * otherwise encounter collisions for hashCodes that do not differ  
    * in lower bits. Note: Null keys always map to hash 0, thus index 0.  
    */  
   static int hash(int h) {  
     // This function ensures that hashCodes that differ only by  
     // constant multiples at each bit position have a bounded  
     // number of collisions (approximately 8 at default load factor).  
     h ^= (h >>> 20) ^ (h >>> 12);  
     return h ^ (h >>> 7) ^ (h >>> 4);  
   }  
   static int indexFor(int h, int length) {  
     return h & (length-1);  
 }  
   public boolean containsValue(Object value) {  
      if (value == null)  
       return containsNullValue();  
      Entry[] tab = table;  
     for (int i = 0; i < tab.length ; i++)  
       for (Entry e = tab[i] ; e != null ; e = e.next)  
         if (value.equals(e.value))  
           return true;  
      return false;  
 }  
   static class Entry<K,V> implements Map.Entry<K,V> {  
     final K key;  
     V value;  
     Entry<K,V> next;  
     final int hash;  
     public final boolean equals(Object o) {  
       if (!(o instanceof Map.Entry))  
         return false;  
       Map.Entry e = (Map.Entry)o;  
       Object k1 = getKey();  
       Object k2 = e.getKey();  
       if (k1 == k2 || (k1 != null && k1.equals(k2))) {  
         Object v1 = getValue();  
         Object v2 = e.getValue();  
         if (v1 == v2 || (v1 != null && v1.equals(v2)))  
           return true;  
       }  
       return false;  
     }  
     public final int hashCode() {  
       return (key==null  ? 0 : key.hashCode()) ^  
           (value==null ? 0 : value.hashCode());  
     }  
   }  
   private abstract class HashIterator<E> implements Iterator<E> {  
     Entry<K,V> next;     // next entry to return  
     int expectedModCount;     // For fast-fail  
     int index;          // current slot  
     Entry<K,V> current;     // current entry  
     HashIterator() {  
       expectedModCount = modCount;  
       if (size > 0) { // advance to first entry  
         Entry[] t = table;  
         while (index < t.length && (next = t[index++]) == null)  
           ;  
       }  
     }  
     public final boolean hasNext() {  
       return next != null;  
     }  
     final Entry<K,V> nextEntry() {  
       if (modCount != expectedModCount)  
         throw new ConcurrentModificationException();  
       Entry<K,V> e = next;  
       if (e == null)  
         throw new NoSuchElementException();  
       if ((next = e.next) == null) {  
         Entry[] t = table;  
         while (index < t.length && (next = t[index++]) == null)  
           ;  
       }  
        current = e;  
       return e;  
     }  
     public void remove() {  
       if (current == null)  
         throw new IllegalStateException();  
       if (modCount != expectedModCount)  
         throw new ConcurrentModificationException();  
       Object k = current.key;  
       current = null;  
       HashMap.this.removeEntryForKey(k);  
       expectedModCount = modCount;  
     }  
   void resize(int newCapacity) {  
     Entry[] oldTable = table;  
     int oldCapacity = oldTable.length;  
     if (oldCapacity == MAXIMUM_CAPACITY) {  
       threshold = Integer.MAX_VALUE;  
       return;  
     }  
     Entry[] newTable = new Entry[newCapacity];  
     transfer(newTable);  
     table = newTable;  
     threshold = (int)(newCapacity * loadFactor);  
   }  
   void transfer(Entry[] newTable) {  
     Entry[] src = table;  
     int newCapacity = newTable.length;  
     for (int j = 0; j < src.length; j++) {  
       Entry<K,V> e = src[j];  
       if (e != null) {  
         src[j] = null;  
         do {  
           Entry<K,V> next = e.next;  
           int i = indexFor(e.hash, newCapacity);  
           e.next = newTable[i];  
           newTable[i] = e;  
           e = next;  
         } while (e != null);  
       }  
     }  
   }  
   private final class ValueIterator extends HashIterator<V> {  
     public V next() {  
       return nextEntry().value;  
     }  
   }  
   private final class KeyIterator extends HashIterator<K> {  
     public K next() {  
       return nextEntry().getKey();  
     }  
   }  
   private final class EntryIterator extends HashIterator<Map.Entry<K,V>> {  
     public Map.Entry<K,V> next() {  
       return nextEntry();  
     }  
   }  
   // Subclass overrides these to alter behavior of views' iterator() method  
   Iterator<K> newKeyIterator()  {  
     return new KeyIterator();  
   }  
   Iterator<V> newValueIterator()  {  
     return new ValueIterator();  
   }  
   Iterator<Map.Entry<K,V>> newEntryIterator()  {  
     return new EntryIterator();  
   }  
   // Views  
   private transient Set<Map.Entry<K,V>> entrySet = null;  
 }  
   public Set<K> keySet() {  
     Set<K> ks = keySet;  
     return (ks != null ? ks : (keySet = new KeySet()));  
   }  
   private final class KeySet extends AbstractSet<K> {  
     public Iterator<K> iterator() {  
       return newKeyIterator();  
     }  
     public int size() {  
       return size;  
     }  
     public boolean contains(Object o) {  
       return containsKey(o);  
     }  
     public boolean remove(Object o) {  
       return HashMap.this.removeEntryForKey(o) != null;  
     }  
     public void clear() {  
       HashMap.this.clear();  
     }  
 }  
   public Collection<V> values() {  
     Collection<V> vs = values;  
     return (vs != null ? vs : (values = new Values()));  
   }  
   private final class Values extends AbstractCollection<V> {  
     public Iterator<V> iterator() {  
       return newValueIterator();  
     }  
     public int size() {  
       return size;  
     }  
     public boolean contains(Object o) {  
       return containsValue(o);  
     }  
     public void clear() {  
       HashMap.this.clear();  
     }  
 }  
   private void writeObject(java.io.ObjectOutputStream s)  
     throws IOException  
   {  
      Iterator<Map.Entry<K,V>> i =  
        (size > 0) ? entrySet0().iterator() : null;  
      // Write out the threshold, loadfactor, and any hidden stuff  
      s.defaultWriteObject();  
      // Write out number of buckets  
      s.writeInt(table.length);  
      // Write out size (number of Mappings)  
      s.writeInt(size);  
     // Write out keys and values (alternating)  
      if (i != null) {  
        while (i.hasNext()) {  
           Map.Entry<K,V> e = i.next();  
           s.writeObject(e.getKey());  
           s.writeObject(e.getValue());  
        }  
     }  
   }  
   private void readObject(java.io.ObjectInputStream s)  
      throws IOException, ClassNotFoundException  
   {  
      // Read in the threshold, loadfactor, and any hidden stuff  
      s.defaultReadObject();  
      // Read in number of buckets and allocate the bucket array;  
      int numBuckets = s.readInt();  
      table = new Entry[numBuckets];  
    init(); // Give subclass a chance to do its thing.  
      // Read in size (number of Mappings)  
      int size = s.readInt();  
      // Read the keys and values, and put the mappings in the HashMap  
      for (int i=0; i<size; i++) {  
        K key = (K) s.readObject();  
        V value = (V) s.readObject();  
        putForCreate(key, value);  
      }  
 }  
 }

Hashtable

 public class Hashtable<K,V>  
   extends Dictionary<K,V>  
 implements Map<K,V>, Cloneable, java.io.Serializable {  
   private transient Entry[] table;  
   private transient int count;  
   private int threshold;  
   private float loadFactor;  
   public Hashtable(int initialCapacity, float loadFactor) {  
      this.loadFactor = loadFactor;  
      table = new Entry[initialCapacity];  
      threshold = (int)(initialCapacity * loadFactor);  
   }  
   public Hashtable(Map<? extends K, ? extends V> t) {  
      this(Math.max(2*t.size(), 11), 0.75f);  
      putAll(t);  
 }  
   public synchronized V get(Object key) {  
      Entry tab[] = table;  
      int hash = key.hashCode();  
      int index = (hash & 0x7FFFFFFF) % tab.length;  
      for (Entry<K,V> e = tab[index] ; e != null ; e = e.next) {  
        if ((e.hash == hash) && e.key.equals(key)) {  
           return e.value;  
        }  
      }  
      return null;  
   }  
   public synchronized V put(K key, V value) {  
      // Make sure the value is not null  
      if (value == null) {  
        throw new NullPointerException();  
      }  
      // Makes sure the key is not already in the hashtable.  
      Entry tab[] = table;  
      int hash = key.hashCode();  
      int index = (hash & 0x7FFFFFFF) % tab.length;  
      for (Entry<K,V> e = tab[index] ; e != null ; e = e.next) {  
        if ((e.hash == hash) && e.key.equals(key)) {  
           V old = e.value;  
           e.value = value;  
           return old;  
        }  
      }  
      modCount++;  
      if (count >= threshold) {  
        // Rehash the table if the threshold is exceeded  
        rehash();  
       tab = table;  
       index = (hash & 0x7FFFFFFF) % tab.length;  
      }  
      // Creates the new entry.  
      Entry<K,V> e = tab[index];  
      tab[index] = new Entry<K,V>(hash, key, value, e);  
      count++;  
      return null;  
 }  
   protected void rehash() {  
      int oldCapacity = table.length;  
      Entry[] oldMap = table;  
      int newCapacity = oldCapacity * 2 + 1;  
      Entry[] newMap = new Entry[newCapacity];  
      modCount++;  
      threshold = (int)(newCapacity * loadFactor);  
      table = newMap;  
      for (int i = oldCapacity ; i-- > 0 ;) {  
        for (Entry<K,V> old = oldMap[i] ; old != null ; ) {  
           Entry<K,V> e = old;  
           old = old.next;  
           int index = (e.hash & 0x7FFFFFFF) % newCapacity;  
           e.next = newMap[index];  
           newMap[index] = e;  
        }  
      }  
 }  
   public synchronized V remove(Object key) {  
      Entry tab[] = table;  
      int hash = key.hashCode();  
      int index = (hash & 0x7FFFFFFF) % tab.length;  
      for (Entry<K,V> e = tab[index], prev = null ; e != null ; prev = e, e = e.next) {  
        if ((e.hash == hash) && e.key.equals(key)) {  
           modCount++;  
           if (prev != null) {  
             prev.next = e.next;  
           } else {  
             tab[index] = e.next;  
           }  
           count--;  
           V oldValue = e.value;  
           e.value = null;  
           return oldValue;  
        }  
      }  
      return null;  
   }  
   public synchronized void putAll(Map<? extends K, ? extends V> t) {  
     for (Map.Entry<? extends K, ? extends V> e : t.entrySet())  
       put(e.getKey(), e.getValue());  
   }  
   public synchronized void clear() {  
      Entry tab[] = table;  
      modCount++;  
      for (int index = tab.length; --index >= 0; )  
        tab[index] = null;  
      count = 0;  
 }  
   public synchronized boolean containsKey(Object key) {  
      Entry tab[] = table;  
      int hash = key.hashCode();  
      int index = (hash & 0x7FFFFFFF) % tab.length;  
      for (Entry<K,V> e = tab[index] ; e != null ; e = e.next) {  
        if ((e.hash == hash) && e.key.equals(key)) {  
           return true;  
        }  
      }  
      return false;  
 }  
 }  
 public abstract class AbstractMap<K,V> implements Map<K,V> {  
   public int hashCode() {  
      int h = 0;  
      Iterator<Entry<K,V>> i = entrySet().iterator();  
      while (i.hasNext())  
        h += i.next().hashCode();  
      return h;  
   }  
    */  
   public boolean equals(Object o) {  
      if (o == this)  
        return true;  
      if (!(o instanceof Map))  
        return false;  
      Map<K,V> m = (Map<K,V>) o;  
      if (m.size() != size())  
        return false;  
     try {  
       Iterator<Entry<K,V>> i = entrySet().iterator();  
       while (i.hasNext()) {  
         Entry<K,V> e = i.next();  
           K key = e.getKey();  
         V value = e.getValue();  
         if (value == null) {  
           if (!(m.get(key)==null && m.containsKey(key)))  
             return false;  
         } else {  
           if (!value.equals(m.get(key)))  
             return false;  
         }  
       }  
     } catch (ClassCastException unused) {  
       return false;  
     } catch (NullPointerException unused) {  
       return false;  
     }  
      return true;  
 }  
 }

LinkedHashMap[TODO]

public class LinkedHashMap extends HashMap implements Map

{

// The head of the doubly linked list.

private transient Entry header;

* The iteration ordering method for this linked hash map: true

* for access-order, false for insertion-order.

private final boolean accessOrder;

void init() {

header = new Entry(-1, null, null, null);

header.before = header.after = header;

}

HashSet

public class HashSet extends AbstractSet implements Set, Cloneable, java.io.Serializable

{

private transient HashMap map;

// Dummy value to associate with an Object in the backing Map

private static final Object PRESENT = new Object();

public HashSet() {map = new HashMap();}

public boolean add(E e) {return map.put(e, PRESENT)==null;}

}

TreeMap

public class TreeMap extends AbstractMap

implements NavigableMap, Cloneable, java.io.Serializable

// it does not extend HashMap

{

private final Comparator comparator;

private transient Entry root = null;

private transient int modCount = 0;

// private Entry getEntry(Object key)

* Returns a Set view of the keys contained in this map. The set's

* iterator will return the keys in ascending order. The map is backed by

* this TreeMap instance, so changes to this map are reflected in

* the Set, and vice-versa. The Set supports element removal, which

* removes the corresponding mapping from the map, via the

* Iterator.remove, Set.remove, removeAll,

* retainAll, and clear operations. It does not support

* the add or addAll operations.

* @return a set view of the keys contained in this TreeMap.

public Set keySet() {

if (keySet == null) {

keySet = new AbstractSet() {

public Iterator iterator() {

return new KeyIterator();

}

public int size() {

return TreeMap.this.size();

}

public boolean contains(Object o) {

return containsKey(o);

}

public boolean remove(Object o) {

int oldSize = size;

TreeMap.this.remove(o);

return size != oldSize;

}

public void clear() {

TreeMap.this.clear();

}

};

}

return keySet;

}

public Collection values() {

if (values == null) {

values = new AbstractCollection() {

public Iterator iterator() {

return new ValueIterator();

}

public int size() {

return TreeMap.this.size();

}

public boolean contains(Object o) {

for (Entry e = firstEntry(); e != null; e = successor(e))

if (valEquals(e.getValue(), o))

return true;

return false;

}

public boolean remove(Object o) {

for (Entry e = firstEntry(); e != null; e = successor(e)) {

if (valEquals(e.getValue(), o)) {

deleteEntry(e);

return true;

}

return false;

}

public void clear() {

TreeMap.this.clear();

}

};

}

return values;

}

public boolean containsValue(Object value) {

return (root==null ? false :

(value==null ? valueSearchNull(root)

: valueSearchNonNull(root, value)));

}

private boolean valueSearchNull(Entry n) {

if (n.value == null)

return true;

// Check left and right subtrees for value

return (n.left != null && valueSearchNull(n.left)) ||

(n.right != null && valueSearchNull(n.right));

}

private boolean valueSearchNonNull(Entry n, Object value) {

// Check this node for the value

if (value.equals(n.value))

return true;

// Check left and right subtrees for value

return (n.left != null && valueSearchNonNull(n.left, value)) ||

(n.right != null && valueSearchNonNull(n.right, value));

}

static final class Entry implements Map.Entry {

K key;

V value;

Entry left = null;

Entry right = null;

Entry parent;

boolean color = BLACK;

}

// Returns the first Entry in the TreeMap (according to the TreeMap's

// key-sort function). Returns null if the TreeMap is empty.

final Entry getFirstEntry() {

Entry p = root;

if (p != null)

while (p.left != null)

p = p.left;

return p;

}

public boolean containsValue(Object value) {

for (Entry e = getFirstEntry(); e != null; e = successor(e))

if (valEquals(value, e.value))

return true;

return false;

}

//final Entry getLastEntry()

// Returns the successor of the specified Entry, or null if no such.

static TreeMap.Entry successor(Entry t) {

if (t == null)

return null;

else if (t.right != null) {

Entry p = t.right;

while (p.left != null)

p = p.left;

return p;

} else {

Entry p = t.parent;

Entry ch = t;

while (p != null && ch == p.right) {

ch = p;

p = p.parent;

}

return p;

}

// Returns the predecessor of the specified Entry, or null if no such.

static Entry predecessor(Entry t) {

if (t == null)

return null;

else if (t.left != null) {

Entry p = t.left;

while (p.right != null)

p = p.right;

return p;

} else {

Entry p = t.parent;

Entry ch = t;

while (p != null && ch == p.left) {

ch = p;

p = p.parent;

}

return p;

}

[TODO]

How BlockingQueue and ConcurrentHashMap are implemented?

ThreadLocal

public class ThreadLocal {

public T get() {

Thread t = Thread.currentThread();

ThreadLocalMap map = getMap(t);

if (map != null) {

ThreadLocalMap.Entry e = map.getEntry(this);

if (e != null)

return (T)e.value;

}

return setInitialValue();

}

public void set(T value) {

Thread t = Thread.currentThread();

ThreadLocalMap map = getMap(t);

if (map != null)

map.set(this, value);

else

createMap(t, value);

}

void createMap(Thread t, T firstValue) {

t.threadLocals = new ThreadLocalMap(this, firstValue);

}

ThreadLocalMap getMap(Thread t) {

return t.threadLocals;

}

English Punctuation

2011-06-01T19:35:00.005-07:00

English Punctuation

?　question mark

!　exclamation mark/exclamation point

/　slash, divide, oblique

\　backslash, sometimes escape

//　slash-slash, comment

&　ampersand, and, reference, ref

*　asterisk, multiply, star, pointer

~　tilde　波浪符

'　apostrophe　撇号

&　ampersand 和号

"　single quotation marks

""　double quotation marks

→　arrow

...　ellipsis/dots

{　open brace, open curly

}　close brace, close curly

(　open parenthesis, open paren

)　close parenthesis, close paren

() parentheses/brakets

[　open bracket

]　close bracket

[] square brackets

.　period, dot

|　vertical bar, vertical virgule

#　pound　井号

.　full stop

,　comma

:　colon

;　semicolon

-　hyphen

-- dash

‖ parallel

～　swung dash 代字号

§　section; division 分节号

+　 plus　sign

-　 minus sign,

±　plus or minus sign

×　multiple sign, product sign

÷　division sign

＝　equality sign

≠　Not equality sign

≡　equivalent to

≌　equal to or approximately equal to

≈　approximately equal to

＜　less than sign

＞　more than sign

≮　not less than sign

≯　not more than sign

≤　less than or equal to

≥　more than or equal to

%　 per cent

‰　per mill

-∞　infinity

+∞　infinity

∝　varies as

√　(square) root

∵　since; because

∴　hence

∷　equals, as (proportion)

∠　angle

⌒　semicircle

⊙　circle

○　circumference

π　pi 圆周率

△　triangle

⊥　perpendicular to　垂直于

∪　union of

∩　intersection of

∫　the integral of …的积分

∑　(sigma) summation of　总和

°　degree

′　minute

″　second

℃　Celsius system

Resources:

http://www.hao360.com/page/details.asp?id=6595

Java New Features History

2011-06-01T03:36:00.001-07:00

Java New Features History

JDK 5 New Features

Generics

Annotations(Metadata)

Autoboxing/Unboxing

Typesafe Enums

Enhanced for Loop

Varargs

Static Import

JDK 6 New Features

JDK 7 New Features

Language Changes

Strings in switch

Multiple Exception Handling

Elvis Operator

Standard example:

String s = mayBeNull?.toString() ?: "null";

Auto-unboxing example:

Integer ival = ...; // may be null

int i = ival ?: -1; // no NPE from unboxing

JSR 292: Support for dynamically-typed languages

NIO.2

TLS 1.2

JDBC 4.1

Support Unicode 6.0

New File I/O API

Asynchronous I/0