Geekality » Java

Java: InputStream line iterator

Torleif — Mon, 23 May 2011 18:09:11 +0000

Wanted to create an easy interface for reading lines from a stream. It should take care of all the annoying Java IO nitty-gritty for me and I wanted to use it simply by throwing it into a for loop.

Found some pieces of code here and there, added some of my own and ended up with a LineReader class. Works as far as I can see, but let me know if you test it out and you find any bugs

import java.io.BufferedReader;
import java.io.Closeable;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
import java.util.NoSuchElementException;

/**
* Represents the lines found in an {@link InputStream}. The lines are read
* one at a time using {@link BufferedReader#readLine()} and may be streamed
* through an iterator or returned all at once.
*
*

This class does not handle any concurrency issues.
*
*

The stream is closed automatically when the for loop is done :)
*
*

{@code

 * for(String line : new LineReader(stream))

 *      // ...

 * }

*
*

An {@link IllegalStateException} will be thrown if any {@link IOException}s
* occur when reading or closing the stream.
*
* @author Torleif Berger
* @license http://creativecommons.org/licenses/by/3.0/
* @see http://www.geekality.net/?p=1614
*/
public class LineReader implements Iterable<String>, Closeable
{
private BufferedReader reader;

public LineReader(InputStream stream)
{
reader = new BufferedReader(new InputStreamReader(stream));
}

/**
* Closes the underlying stream.
*/
@Override
public void close() throws IOException
{
reader.close();
}

/**
* Makes sure the underlying stream is closed.
*/
@Override
protected void finalize() throws Throwable
{
close();
}

/**
* Returns an iterator over the lines remaining to be read.
*
*

The underlying stream is closed automatically once {@link Iterator#hasNext()}
* returns false. This means that the stream should be closed after using a for loop.
*
* @return This iterator.
*/
@Override
public Iterator<String> iterator()
{
return new LineIterator();
}

/**
* Returns all lines remaining to be read and closes the stream.
*
* @return The lines read from the stream.
*/
public Collection<String> readLines()
{
Collection<String> lines = new ArrayList<String>();
for(String line : this)
{
lines.add(line);
}
return lines;
}

private class LineIterator implements Iterator<String>
{
private String nextLine;

public String bufferNext()
{
try
{
return nextLine = reader.readLine();
}
catch (IOException e)
{
throw new IllegalStateException("I/O error while reading stream.", e);
}
}

public boolean hasNext()
{
boolean hasNext = nextLine != null || bufferNext() != null;

if ( ! hasNext)
try
{
reader.close();
}
catch (IOException e)
{
throw new IllegalStateException("I/O error when closing stream.", e);
}

return hasNext;
}

public String next()
{
if ( ! hasNext())
throw new NoSuchElementException();

String result = nextLine;
nextLine = null;
return result;
}

public void remove() {
throw new UnsupportedOperationException();
}
}
}

You can use it in a streaming fashion:

for(String line : new LineReader(stream))
{
// ...
}

Or grab all the lines at once:

Collection<String> lines = new LineReader(stream).readLines();

Let me know what you think! Feedback is welcome as always

Java: How to deal with the BOM in a Unicode InputStream

Torleif — Mon, 23 May 2011 17:49:16 +0000

Ok, so I was happily reading CSV files from an SFTP server. The file content is returned as an InputStream and I I used a BufferedReader to read it line by line. Each line contained either a header or an order. The header lines started with the string “HDR”.

However, I suddenly discovered that my code was consistently skipping the first header (and as a result the orders belonging to it). The reason, I found, was simple. The first header, on the first line, didn’t start with “HDR”, it started with “□HDR”! And that undisplayable square turned out to be a Unicode Byte Order Mark (BOM).

To deal with the BOM, we can use a simple class I found at StackOverflow which handles it for us. Here it is

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298

import java.io.IOException;
import java.io.InputStream;
import java.io.PushbackInputStream;

/**
* The {@link UnicodeBOMInputStream} class wraps any
* {@link InputStream} and detects the presence of any Unicode BOM
* (Byte Order Mark) at its beginning, as defined by
* RFC 3629 - UTF-8, a transformation format of ISO 10646
*
*

The
* Unicode FAQ
* defines 5 types of BOMs:

```
00 00 FE FF  = UTF-32, big-endian
```

```
FF FE 00 00  = UTF-32, little-endian
```

```
FE FF        = UTF-16, big-endian
```

FF FE        = UTF-16, little-endian

```
EF BB BF     = UTF-8
```

*
*

Use the {@link #getBOM()} method to know whether a BOM has been detected
* or not.
*

Use the {@link #skipBOM()} method to remove the detected BOM from the
* wrapped {@link InputStream} object.

*
* @author Gregory Pakosz
* @see http://stackoverflow.com/q/1835430/39321#1835529
*/
class UnicodeBOMInputStream extends InputStream
{
/**
* Type safe enumeration class that describes the different types of Unicode
* BOMs.
*/
public static final class BOM
{
/**
* NONE.
*/
public static final BOM NONE = new BOM(new byte[]{},"NONE");

/**
* UTF-8 BOM (EF BB BF).
*/
public static final BOM UTF_8 = new BOM(new byte[]{(byte)0xEF,
(byte)0xBB,
(byte)0xBF},
"UTF-8");

/**
* UTF-16, little-endian (FF FE).
*/
public static final BOM UTF_16_LE = new BOM(new byte[]{ (byte)0xFF,
(byte)0xFE},
"UTF-16 little-endian");

/**
* UTF-16, big-endian (FE FF).
*/
public static final BOM UTF_16_BE = new BOM(new byte[]{ (byte)0xFE,
(byte)0xFF},
"UTF-16 big-endian");

/**
* UTF-32, little-endian (FF FE 00 00).
*/
public static final BOM UTF_32_LE = new BOM(new byte[]{ (byte)0xFF,
(byte)0xFE,
(byte)0x00,
(byte)0x00},
"UTF-32 little-endian");

/**
* UTF-32, big-endian (00 00 FE FF).
*/
public static final BOM UTF_32_BE = new BOM(new byte[]{ (byte)0x00,
(byte)0x00,
(byte)0xFE,
(byte)0xFF},
"UTF-32 big-endian");

/**
* Returns a {@link String} representation of this {@link BOM}.
* value.
*/
public final String toString()
{
return description;
}

/**
* Returns the bytes corresponding to this {@link BOM} value.
*/
public final byte[] getBytes()
{
final int length = bytes.length;
final byte[] result = new byte[length];

// Make a defensive copy
System.arraycopy(bytes,0,result,0,length);

return result;
}

private BOM(final byte bom[], final String description)
{
this.bytes = bom;
this.description = description;
}

final byte bytes[];
private final String description;

}

/**
* Constructs a new {@link UnicodeBOMInputStream} that wraps the
* specified {@link InputStream}.
*
* @param inputStream an {@link InputStream}.
*
* @throws IOException on reading from the specified {@link InputStream}
* when trying to detect the Unicode BOM.
*/
public UnicodeBOMInputStream(final InputStream inputStream) throws IOException

{
in = new PushbackInputStream(inputStream,4);

final byte bom[] = new byte[4];
final int read = in.read(bom);

switch(read)
{
case 4:
if ((bom[0] == (byte)0xFF) &&
(bom[1] == (byte)0xFE) &&
(bom[2] == (byte)0x00) &&
(bom[3] == (byte)0x00))
{
this.bom = BOM.UTF_32_LE;
break;
}
else
if ((bom[0] == (byte)0x00) &&
(bom[1] == (byte)0x00) &&
(bom[2] == (byte)0xFE) &&
(bom[3] == (byte)0xFF))
{
this.bom = BOM.UTF_32_BE;
break;
}

case 3:
if ((bom[0] == (byte)0xEF) &&
(bom[1] == (byte)0xBB) &&
(bom[2] == (byte)0xBF))
{
this.bom = BOM.UTF_8;
break;
}

case 2:
if ((bom[0] == (byte)0xFF) &&
(bom[1] == (byte)0xFE))
{
this.bom = BOM.UTF_16_LE;
break;
}
else
if ((bom[0] == (byte)0xFE) &&
(bom[1] == (byte)0xFF))
{
this.bom = BOM.UTF_16_BE;
break;
}

default:
this.bom = BOM.NONE;
break;
}

if (read > 0)
in.unread(bom,0,read);
}

/**
* Returns the {@link BOM} that was detected in the wrapped
* {@link InputStream} object.
*
* @return a {@link BOM} value.
*/
public final BOM getBOM()
{
// BOM type is immutable.
return bom;
}

/**
* Skips the {@link BOM} that was found in the wrapped
* {@link InputStream} object.
*
* @return this {@link UnicodeBOMInputStream}.
*
* @throws IOException when trying to skip the BOM from the wrapped {@link InputStream} object.
*/
public final synchronized UnicodeBOMInputStream skipBOM() throws IOException
{
if ( ! skipped)
{
in.skip(bom.bytes.length);
skipped = true;
}
return this;
}

/**
* {@inheritDoc}
*/
public int read() throws IOException
{
return in.read();
}

/**
* {@inheritDoc}
*/
public int read(final byte b[]) throws IOException,
NullPointerException
{
return in.read(b,0,b.length);
}

/**
* {@inheritDoc}
*/
public int read(final byte b[],
final int off,
final int len) throws IOException,
NullPointerException
{
return in.read(b,off,len);
}

/**
* {@inheritDoc}
*/
public long skip(final long n) throws IOException
{
return in.skip(n);
}

/**
* {@inheritDoc}
*/
public int available() throws IOException
{
return in.available();
}

/**
* {@inheritDoc}
*/
public void close() throws IOException
{
in.close();
}

/**
* {@inheritDoc}
*/
public synchronized void mark(final int readlimit)
{
in.mark(readlimit);
}

/**
* {@inheritDoc}
*/
public synchronized void reset() throws IOException
{
in.reset();
}

/**
* {@inheritDoc}
*/
public boolean markSupported()
{
return in.markSupported();
}

private final PushbackInputStream in;
private final BOM bom;
private boolean skipped = false;

}

Looks long, but it’s simple to use.

InputStream cleanStream = new UnicodeBOMInputStream(stream).skipBOM();

When reading the stream, remember to also use an InputStreamReader with the correct character set. For example something like this:

BufferedReader reader = new BufferedReader(new InputStreamReader(cleanStream, "UTF-8"));
String line = null;
while((line = reader.readLine()) != null)
System.out.println(line);
reader.close();

If the file might be any of the Unicode kinds, you can probably use the getBOM method of the UnicodeBOMInputStream to choose the right one.

To sum up, this is tricky and annoying stuff! Please, if something in this post is wrong or inaccurate or should be done differently, please leave a comment. Want to get this right

Found an alternative to the above class in the Apache Commons IO library. It’s called BOMInputStream and seems to do pretty much the same thing, except it has some extra features as well. Might be nice to use if you’re already using that library or don’t mind adding another.

How to set JAVA_HOME and where is whereis on Solaris?

Torleif — Wed, 02 Feb 2011 16:41:50 +0000

To set the JAVA_HOME environment variable you first need to find where your Java installation is located and then set it somehow.

How to find it

$ whereis java
/usr/bin/java
$ ls -l /usr/bin/java
lrwxrwxrwx ... /usr/bin/java -> ../java/bin/java

First command figures out where the java command is located. We then use ls -l and finds that /usr/bin/java actually points to ../java/bin/java. As an absolute path, that would be /usr/java/bin/java. This means our JAVA_HOME should be set to /usr/java.

(I have assumed that Java actually is installed. If it isn’t, you need to do that first)

But where is whereis?

The above example is how it normally goes for me on for example Ubuntu (although the directory you get might be different). However, this is what happened when I did it on a machine running SunOS 5.10:

$ whereis java
-bash: whereis: command not found

Turns out that command doesn’t exist on Solaris. Luckily we can use a command called which instead, which gives you the same result.

$ which java
/usr/bin/java

But I want to use whereis!

Well, turns out you can! Apparently the command actually does exist. It’s just not in your path by default. This command (and some others) can be found in /usr/ucb, so we just need to add that to our PATH and we can use it like normal.

How to set it

There are various ways of doing this, and in my case the simplest way was to add the lines below to a file called ~/.profile.

JAVA_HOME=/usr/java
export JAVA_HOME

PATH=$PATH:/usr/ucb
export PATH

The last two lines shows how to add /usr/ucb to your path, which you can skip if you don’t feel a great need to use whereis later.

Test-Driven Development: By Example

Torleif — Tue, 01 Dec 2009 17:36:16 +0000

Book cover

I earlier wrote about the book, The Art of Unit Testing, which I finished a while ago. That book was very good and was focused on how to write good unit tests. It also mentioned Test-Driven Development, TDD, but not too much. The book I read next, which I finished a few days ago, was kind of the other way around. Pretty much only about TDD. And from the title, Test-Driven Development: By Example, that shouldn’t be much of a shocker

The book is written by Kent Beck and explains the basics about TDD. It does this pretty well, step by step, by example, just like the title says. The book also talks some about TDD in a more general sense. What it is, how it works, why it works, et cetera.

From the back-cover:

Readers will learn to:

Solve complicated tasks, beginning with the simple and proceeding to the more complex.

Write automated tests before coding.

Grow a design organically by refactoring to add design decisions one at a time.

Create tests for more complicated logic, including reflection and exceptions.

Use patterns to decide what tests to write.

Create tests using xUnit, the architecture at the heart of many programmer-oriented testing tools.

It pretty much kept the promises The only part I didn’t really like about this book was that I found much of the coding kind of messy. Could be because the languages used in the examples are Java and Python, which I don’t really like that much syntax-wise. An even more important reason though, is that I just finished reading The Art of Unit Testing So, I suppose I have become a bit demanding when it comes to cleanliness when coding and writing unit-tests. I suppose that is a good thing (I do however not claim in any way that the code I currently write is extremely clean in any way! But I do strive for it to be so ;)

I can really recommend this book to anyone who are curious to get familiar with TDD. I certainly see the value of doing TDD and is in the process of trying to incorporate it into my development habits.

Red — Write a little test that doesn’t work, and perhaps doesn’t even compile at first.

Green — Make the test work quicky, committing whatever sins necessary in the process.

Refactor — Eliminate all of the duplication created in merely getting the test to work.

Red/green/refactor — the TDD mantra.

You can get it at Amazon, and probably a lot of other places too as it is a pretty well-known book.