The Slow Killers: Dust, Humidity, and Corrosion

The Environment Is Part of the Machine

An air cooled miner pulls hundreds of cubic feet of outside air through itself every minute. Whatever is in that air passes over the boards: dust, pollen, insects, moisture, agricultural chemicals, road salt aerosols. Traditional data centers usually condition and filter air before it reaches a server. Mining economics push many sites toward raw outside air, which can work well, and it also means the quality of that air becomes part of how the machines age.

Dust

Dust affects a miner in two main ways. It can blanket heatsinks and boards and act as insulation, so chips run hotter at the same load. It can also load fan bearings, so fans wear faster. The progression is often gradual: intake temperatures look fine, board temperatures creep up, fans run harder, and then temperature faults and fan lost errors appear in what looks like a sudden wave. Here is what those Antminer error codes mean.

Approaches operators commonly use include:

• Intake filtration. Many sites use pleated filter media on the intake wall. Filter sizing involves a trade off: finer or smaller filters catch more dust but restrict more airflow, and a clogged filter can raise intake temperatures. Many operators change filters on a schedule or based on pressure readings.
• Seasonal awareness. Dust load changes with the surroundings. A site next to farmland may see harvest dust seasons, a site on a gravel road may see traffic dust, and cottonwood season can load a filter wall within a couple of weeks.
• Cleaning methods. A common method is compressed air at controlled pressure with the machine powered down, often outside the hall. Cleaning a running machine can push dust deeper and may damage fan blades.

Humidity and Condensation

Moisture affects boards mainly through condensation and corrosion. The mechanism is simple: warm humid air touching a cooler surface drops its moisture as liquid, in this case right on the electronics. It can happen when machines cycle off and cool down in humid air, when spring weather swings warm and wet after a cold night, or at sites that pull in saturated air. D-Central, a mining equipment company, lists Antminer operating conditions of 10 to 90 percent relative humidity, non condensing. Some cold weather sites mix warm exhaust air back into the intake in winter to keep the air warm and dry. That is designed recirculation, an intentional exception to the usual approach of keeping hot air out of the cold aisle.

Corrosion is the aftermath. Moisture combined with contaminants can attack solder joints and traces, and boards may begin dropping chips, which shows up as creeping partial chip counts in the logs. Visible green or white residue on boards can indicate prolonged exposure. When that exposure began, and where, is often examined using logs and other records.

How Environmental Records Are Used

Because corrosion develops slowly, timing is often a question when damaged machines come out of a hosting site. Several kinds of records can bear on it: machine logs that show when chip counts began to decline, intake temperature and humidity data, filter change logs, cleaning tickets, and the physical condition of the boards. Each has limits. Sensors can be poorly placed or drift, logs can be overwritten, and corrosion can have more than one cause, including conditions during shipping or storage before a machine arrived at a site.

• Environmental logging. Many sites record temperature and humidity at the intakes and archive the data. How useful that data is depends on sensor placement, calibration, and how long the records are kept.
• Contract terms. Hosting agreements vary. Some address filtration, humidity management, and cleaning cadence in detail, while others say little about environmental conditions.

For more on how these topics come up in hosting matters, see the ASIC hosting expert witness page. For the kinds of records that come up in mining disputes, see this post on mining documentation.