我已经用Java构建了一个图像分类器，我想根据这里提供的图像进行测试：http://yann.lecun.com/exdb/mnist/
不幸的是，如果您下载train-images-idx3-ubyte.gz或任何其他3个文件，它们都是文件类型：.idx1-ubyte
第一个问题： 我想知道是否有人可以给我关于如何将.idx1-ubyte变成bitmaps（.bmp）文件的说明？
第二个问题： 或者我一般如何阅读这些文件？
有关IDX文件格式的信息： IDX文件格式是各种数值类型的向量和多维矩阵的简单格式。 基本格式为：
magic number
size in dimension 0
size in dimension 1
size in dimension 2
.....
size in dimension N
data
幻数是一个整数（MSB优先）。前2个字节始终为0.
第三个字节编码数据类型：
0x08: unsigned byte
0x09: signed byte
0x0B: short (2 bytes)
0x0C: int (4 bytes)
0x0D: float (4 bytes)
0x0E: double (8 bytes)
第4个字节编码向量/矩阵的维数：向量为1，矩阵为2 ....
每个维度的大小为4字节整数（MSB优先，高端，与大多数非英特尔处理器一样）。
数据存储在C数组中，即最后一维中的索引变化最快。
答案 0 :(得分：10)
相当直截了当，正如WPrecht所说：“URL描述了你必须解码的格式”。这是我的idx文件的ImageSet导出器，不是很干净，但它做了它必须做的事情。
public class IdxReader {
public static void main(String[] args) {
// TODO Auto-generated method stub
FileInputStream inImage = null;
FileInputStream inLabel = null;
String inputImagePath = "CBIR_Project/imagesRaw/MNIST/train-images-idx3-ubyte";
String inputLabelPath = "CBIR_Project/imagesRaw/MNIST/train-labels-idx1-ubyte";
String outputPath = "CBIR_Project/images/MNIST_Database_ARGB/";
int[] hashMap = new int[10];
try {
inImage = new FileInputStream(inputImagePath);
inLabel = new FileInputStream(inputLabelPath);
int magicNumberImages = (inImage.read() << 24) | (inImage.read() << 16) | (inImage.read() << 8) | (inImage.read());
int numberOfImages = (inImage.read() << 24) | (inImage.read() << 16) | (inImage.read() << 8) | (inImage.read());
int numberOfRows = (inImage.read() << 24) | (inImage.read() << 16) | (inImage.read() << 8) | (inImage.read());
int numberOfColumns = (inImage.read() << 24) | (inImage.read() << 16) | (inImage.read() << 8) | (inImage.read());
int magicNumberLabels = (inLabel.read() << 24) | (inLabel.read() << 16) | (inLabel.read() << 8) | (inLabel.read());
int numberOfLabels = (inLabel.read() << 24) | (inLabel.read() << 16) | (inLabel.read() << 8) | (inLabel.read());
BufferedImage image = new BufferedImage(numberOfColumns, numberOfRows, BufferedImage.TYPE_INT_ARGB);
int numberOfPixels = numberOfRows * numberOfColumns;
int[] imgPixels = new int[numberOfPixels];
for(int i = 0; i < numberOfImages; i++) {
if(i % 100 == 0) {System.out.println("Number of images extracted: " + i);}
for(int p = 0; p < numberOfPixels; p++) {
int gray = 255 - inImage.read();
imgPixels[p] = 0xFF000000 | (gray<<16) | (gray<<8) | gray;
}
image.setRGB(0, 0, numberOfColumns, numberOfRows, imgPixels, 0, numberOfColumns);
int label = inLabel.read();
hashMap[label]++;
File outputfile = new File(outputPath + label + "_0" + hashMap[label] + ".png");
ImageIO.write(image, "png", outputfile);
}
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
if (inImage != null) {
try {
inImage.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
if (inLabel != null) {
try {
inLabel.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
}
答案 1 :(得分：0)
网址描述了您必须解码的格式，他们提到它是非标准的，因此明显的Google搜索不会显示任何使用代码。然而，它是非常直接的标题后面跟着一个0-255灰度值的28x28像素矩阵。
一旦您读出数据（记得注意字节顺序），创建BMP文件就是直截了当的。
我向您推荐以下文章：
How to make bmp image from pixel byte array in java
他们的问题是关于颜色，但是他们的代码已经适用于灰度级，而这正是你所需要的，你应该能够从那个代码片段获得一些东西。
答案 2 :(得分：0)
我创建了一些用Java读取the MNIST handwritten digits data set的类。这些类可以从下载站点上的可用文件中进行解压缩（解压缩）后读取文件。允许读取原始（压缩）文件的类是小型MnistReader项目的一部分。
下面的这些类是独立（意味着它们不依赖于第三方库），并且基本上在Public Domain中-意味着它们可以被复制到自己的项目中。 （可以提供归因，但不是必需的）：
MnistDecompressedReader类：
import java.io.DataInputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Path;
import java.util.Objects;
import java.util.function.Consumer;
/**
* A class for reading the MNIST data set from the <b>decompressed</b>
* (unzipped) files that are published at
* <a href="http://yann.lecun.com/exdb/mnist/">
* http://yann.lecun.com/exdb/mnist/</a>.
*/
public class MnistDecompressedReader
{
/**
* Default constructor
*/
public MnistDecompressedReader()
{
// Default constructor
}
/**
* Read the MNIST training data from the given directory. The data is
* assumed to be located in files with their default names,
* <b>decompressed</b> from the original files:
* extension) :
* <code>train-images.idx3-ubyte</code> and
* <code>train-labels.idx1-ubyte</code>.
*
* @param inputDirectoryPath The input directory
* @param consumer The consumer that will receive the resulting
* {@link MnistEntry} instances
* @throws IOException If an IO error occurs
*/
public void readDecompressedTraining(Path inputDirectoryPath,
Consumer<? super MnistEntry> consumer) throws IOException
{
String trainImagesFileName = "train-images.idx3-ubyte";
String trainLabelsFileName = "train-labels.idx1-ubyte";
Path imagesFilePath = inputDirectoryPath.resolve(trainImagesFileName);
Path labelsFilePath = inputDirectoryPath.resolve(trainLabelsFileName);
readDecompressed(imagesFilePath, labelsFilePath, consumer);
}
/**
* Read the MNIST training data from the given directory. The data is
* assumed to be located in files with their default names,
* <b>decompressed</b> from the original files:
* extension) :
* <code>t10k-images.idx3-ubyte</code> and
* <code>t10k-labels.idx1-ubyte</code>.
*
* @param inputDirectoryPath The input directory
* @param consumer The consumer that will receive the resulting
* {@link MnistEntry} instances
* @throws IOException If an IO error occurs
*/
public void readDecompressedTesting(Path inputDirectoryPath,
Consumer<? super MnistEntry> consumer) throws IOException
{
String testImagesFileName = "t10k-images.idx3-ubyte";
String testLabelsFileName = "t10k-labels.idx1-ubyte";
Path imagesFilePath = inputDirectoryPath.resolve(testImagesFileName);
Path labelsFilePath = inputDirectoryPath.resolve(testLabelsFileName);
readDecompressed(imagesFilePath, labelsFilePath, consumer);
}
/**
* Read the MNIST data from the specified (decompressed) files.
*
* @param imagesFilePath The path of the images file
* @param labelsFilePath The path of the labels file
* @param consumer The consumer that will receive the resulting
* {@link MnistEntry} instances
* @throws IOException If an IO error occurs
*/
public void readDecompressed(Path imagesFilePath, Path labelsFilePath,
Consumer<? super MnistEntry> consumer) throws IOException
{
try (InputStream decompressedImagesInputStream =
new FileInputStream(imagesFilePath.toFile());
InputStream decompressedLabelsInputStream =
new FileInputStream(labelsFilePath.toFile()))
{
readDecompressed(
decompressedImagesInputStream,
decompressedLabelsInputStream,
consumer);
}
}
/**
* Read the MNIST data from the given (decompressed) input streams.
* The caller is responsible for closing the given streams.
*
* @param decompressedImagesInputStream The decompressed input stream
* containing the image data
* @param decompressedLabelsInputStream The decompressed input stream
* containing the label data
* @param consumer The consumer that will receive the resulting
* {@link MnistEntry} instances
* @throws IOException If an IO error occurs
*/
public void readDecompressed(
InputStream decompressedImagesInputStream,
InputStream decompressedLabelsInputStream,
Consumer<? super MnistEntry> consumer) throws IOException
{
Objects.requireNonNull(consumer, "The consumer may not be null");
DataInputStream imagesDataInputStream =
new DataInputStream(decompressedImagesInputStream);
DataInputStream labelsDataInputStream =
new DataInputStream(decompressedLabelsInputStream);
int magicImages = imagesDataInputStream.readInt();
if (magicImages != 0x803)
{
throw new IOException("Expected magic header of 0x803 "
+ "for images, but found " + magicImages);
}
int magicLabels = labelsDataInputStream.readInt();
if (magicLabels != 0x801)
{
throw new IOException("Expected magic header of 0x801 "
+ "for labels, but found " + magicLabels);
}
int numberOfImages = imagesDataInputStream.readInt();
int numberOfLabels = labelsDataInputStream.readInt();
if (numberOfImages != numberOfLabels)
{
throw new IOException("Found " + numberOfImages
+ " images but " + numberOfLabels + " labels");
}
int numRows = imagesDataInputStream.readInt();
int numCols = imagesDataInputStream.readInt();
for (int n = 0; n < numberOfImages; n++)
{
byte label = labelsDataInputStream.readByte();
byte imageData[] = new byte[numRows * numCols];
read(imagesDataInputStream, imageData);
MnistEntry mnistEntry = new MnistEntry(
n, label, numRows, numCols, imageData);
consumer.accept(mnistEntry);
}
}
/**
* Read bytes from the given input stream, filling the given array
*
* @param inputStream The input stream
* @param data The array to be filled
* @throws IOException If the input stream does not contain enough bytes
* to fill the array, or any other IO error occurs
*/
private static void read(InputStream inputStream, byte data[])
throws IOException
{
int offset = 0;
while (true)
{
int read = inputStream.read(
data, offset, data.length - offset);
if (read < 0)
{
break;
}
offset += read;
if (offset == data.length)
{
return;
}
}
throw new IOException("Tried to read " + data.length
+ " bytes, but only found " + offset);
}
}
MnistEntry类：
import java.awt.image.BufferedImage;
import java.awt.image.DataBuffer;
import java.awt.image.DataBufferByte;
/**
* An entry of the MNIST data set. Instances of this class will be passed
* to the consumer that is given to the {@link MnistCompressedReader} and
* {@link MnistDecompressedReader} reading methods.
*/
public class MnistEntry
{
/**
* The index of the entry
*/
private final int index;
/**
* The class label of the entry
*/
private final byte label;
/**
* The number of rows of the image data
*/
private final int numRows;
/**
* The number of columns of the image data
*/
private final int numCols;
/**
* The image data
*/
private final byte[] imageData;
/**
* Default constructor
*
* @param index The index
* @param label The label
* @param numRows The number of rows
* @param numCols The number of columns
* @param imageData The image data
*/
MnistEntry(int index, byte label, int numRows, int numCols,
byte[] imageData)
{
this.index = index;
this.label = label;
this.numRows = numRows;
this.numCols = numCols;
this.imageData = imageData;
}
/**
* Returns the index of the entry
*
* @return The index
*/
public int getIndex()
{
return index;
}
/**
* Returns the class label of the entry. This is a value in [0,9],
* indicating which digit is shown in the entry
*
* @return The class label
*/
public byte getLabel()
{
return label;
}
/**
* Returns the number of rows of the image data.
* This will usually be 28.
*
* @return The number of rows
*/
public int getNumRows()
{
return numRows;
}
/**
* Returns the number of columns of the image data.
* This will usually be 28.
*
* @return The number of columns
*/
public int getNumCols()
{
return numCols;
}
/**
* Returns a <i>reference</i> to the image data. This will be an array
* of length <code>numRows * numCols</code>, containing values
* in [0,255] indicating the brightness of the pixels.
*
* @return The image data
*/
public byte[] getImageData()
{
return imageData;
}
/**
* Creates a new buffered image from the image data that is stored
* in this entry.
*
* @return The image
*/
public BufferedImage createImage()
{
BufferedImage image = new BufferedImage(getNumCols(),
getNumRows(), BufferedImage.TYPE_BYTE_GRAY);
DataBuffer dataBuffer = image.getRaster().getDataBuffer();
DataBufferByte dataBufferByte = (DataBufferByte) dataBuffer;
byte data[] = dataBufferByte.getData();
System.arraycopy(getImageData(), 0, data, 0, data.length);
return image;
}
@Override
public String toString()
{
String indexString = String.format("%05d", index);
return "MnistEntry["
+ "index=" + indexString + ","
+ "label=" + label + "]";
}
}
阅读器可用于读取未压缩的文件。结果将是MnistEntry个实例，这些实例将传递给使用者：
MnistDecompressedReader mnistReader = new MnistDecompressedReader();
mnistReader.readDecompressedTraining(Paths.get("./data"), mnistEntry ->
{
System.out.println("Read entry " + mnistEntry);
BufferedImage image = mnistEntry.createImage();
...
});
MnistReader项目包含几个examples，说明如何使用这些类读取压缩或未压缩的数据，或从MNIST条目生成PNG图像。