2014/07/12

RAID++ and Storage Pools: Leveraging GPT partitions for Asymmetric Media Logical Volumes. Pt 1.

This is an exploration of addressing Storage problems posed by directly connected or (ethernet) networked drives, not for SAN-connected managed disks served by Enterprise Storage Arrays.

The Problem

One of the most important features of the Veritas Logical Volume Manager (LVM) circa 1995 was the ~1MB disk label that contained a full copy of the LVM information of the drive/volume and allowed drives to be renamed by the system of physically shuffled, intentionally or not.

Today we have a standard, courtesy UEFI, for GUID Partition Tables (GPT) of Storage Devices supported by all major Operating Systems. Can this provide similar, or additional capability?

2014/07/10

RAID++ and Storage Pages: We may be asking the wrong questions

The implied contract between Storage Devices, once HDD's only, and systems is a rather weak one.
Storage Devices return blocks of data on a "Best Efforts" basis, failure & error handling are minimalist or non-existent.
There's no implicit contract with the many other components that are now needed to move data off the Storage Device and into Memory, HBA's, cables, adaptors, switches etc. The move to Ethernet and larger Networks compounds the problem: networks are not nearly error-free. This matters when routinely moving around Exabytes and more: errors and failures are guaranteed for any human-scale observation period.

Turning this weak assurance into usable levels of Reliability and Data Durability is currently left to a rather complex set of layers, which can have subtle & undetectable failure modes or in "Recovery" mode, have unusably poor performance and limited or no resilient against additional failures. We need to improve our models to move past current RAID schemes to routinely support thousands of small drives and new Storage Class Memory.

Scaling Storage to Petabyte and Exabyte sized Pools of mixed technologies needs some new thinking.
New mixed technologies now provide us with multiple Price-Size-Performance components, requiring very careful analysis to optimise Systems against owner criteria.

There is no one true balance between DRAM, PCI-Flash, SSD's, fast-HDD, slow-HDD and near-line/off-line HDD or tape and Optical Disk. What there is, is a willingness of an owner to pay. Presumably they have a preference to pay enough, but not significantly more, for their desired or required "performance", either as "response time" latency or "throughput". Very few clients can afford, or need, to store everything in DRAM with some sort of backup system. It's the highest performance and highest priced solution possible, but is only necessary or desirable in very constrained problems.

DRAM is around $10/GB, Flash and SSD about $1/GB and HDD's from $0.04 to $0.30/GB for raw disk.

Here's a possible new contract between Storage Devices and Clients/Systems:
Data is returned Correct, Complete and Verifiable, in whole or part, between the two Endpoints.

2014/06/22

RAID++: So, you cant afford the extra cost of Data Protection at $0.10-$0.20 per GB?

Summary: You and your business probably now depend on computers and smartphones/tablets for most of your daily work and other activities. If you don't pay up-front to protect your data, you'll pay for it many times over at a later date, when, not if, you have a drive fail and lose all data.

When data is $0.20/GB (or even $1/GB) and wages are $35-60/hour and it will take a minimum of 1 day to reconstruct data, more likely a week+, spending a little money up-front for Data Protection seems prudent to me.

The 1987 Berkeley RAID paper was written at a time few people had PC's and storage cost $40,000/GB in current dollars. The economics of swapping space for computation were compelling at the time, nowdays, very few people have even$1,000 invested in Disk Storage, let alone $250,000.

Good desktops or laptops are now available in $500-$1,000 range, with Commodity Drives costing $0.04-$0.10/GB and Enterprise Drives from $0.12-$0.65/GB, and more for high-spec variants. Times are very different: raw prices have fallen 500,000, Bit Error Rates (BER/UBER) are up ~100 times, Mean Time Between Failures (MTBF) have increased 10-100 fold,  raw read/write rates have increased 100-300 times, while access times (rotation & seek) are 2-5 times different. As is estimated disk utilisation: at some point after 2000 the average drive went for 90%-100% full to ~75%, at least for Desktops. This suggests that drives are now "Big Enough" and not a System Constraint, at least not for Capacity. The advent of affordable, large Flash Memory with reasonable read/write speeds and uniform access times has removed one of the big constraints of storage: random I/O per second.

Researching RAID designs, I was surprised by I.T. Professionals and home users alike, that baulk at the cost of reasonable Data Protection, even $100 for a single USB drive plus a 4-drive NAS unit is definitely "too expensive" (<$1,500). Do they have such volumes of data that the cost of extra drives is overwhelming? Or is the data worth so little, or cost so little, that it's not worth protecting?

2014/06/20

Recruiting FAIL: Update with ITCRA documents

I've put on-line the follow-up documents [links below fold] I received from ITCRA on my complaint, lodged in Aug/Sep 2012 and apparently resolved before Christmas that year. I have on record, because almost all my interactions have been via email, that I wasn't informed on the lack of Natural Justice until around six months after I lodged my grievance.

I was given an undertaking that a) ITCRA would write a Case Study from my complaint, in lieu of informing me of their determinations and actions taken, and b) I would be sent a copy.

A year on, Dec 2013, I enquired after the Case Study. My guess from the delay is that it'd never been written, as promised in writing.

It was only in February this year, 2014, that I received that Case Study.
In a separate email I was informed that the Agent had been dismissed over this matter, presumably in late December 2012.

2014/06/19

Recruiting FAIL: The Gap between Promise and Reality

Following from my own problems with recruiters, a mate sent me the following unsatisfying exchange with a "recruiting" agent, from a supposedly specialist firm.

If you're looking for work, you'll be only too familiar with this mechanical approach where they don't even both to look at your CV and confirm there's a hint of a match.

If you're hiring, this is death. The High Priced Agency might send you warm bodies, but they've done less than you'd want, and possibly would like to believe.

This led me to spending some time researching these "experts".

It's a public company in the UK that sells franchises all around the world - they claim 67 locations and 2,500 staff.
To investors, they claim "high value, high margin". They make a motza out of the suckers on both sides of the table - just look at their financials.

Rummaging around in the company and business names registry and you don't find nearly the entities widely claimed on-line. Is that a problem under the Trade Practice Act (1974) now ACL (2010) part of the CCA (2010) or could someone just register those names and sue them for using them? This is why we have "Hungry Jacks" not "Burger King" burgers. Someone didn't register the names across Australia.

Dig even a little and you find stuff you'd rather not know. A formal warning from ACMA over spamming, a Fairwork complaint on unfair dismissal which doesn't just question the judgement of managers, but makes a case for the employee being consistently 'loose with the truth'.

Then there's the reviews by clients... A few "5-star" ratings, some which read like they were purchased from a sweat-shop in India, others that jibe totally with the rest.

Oh, and one of the original principals who's now worth tens of millions of dollars after the 2005 float is best described as "colourful", four wives, lots of bling and a sting operation for cocaine use.


Date: 19 June 2014
Subject: Re: UX Designer contract role
To: recrutier@progressiverecruitment.com
Dear XXXX,
With respect, the problem with most I.T. recruiting agencies is just this - a data mining program trolled the 1000's of CV's that you have online and sent me this "opportunity". If you took the time to read my resume, perhaps schedule a face to face interview, get to know me, my skills, my strengths and weaknesses, shared a cup of tea, a Skype meeting, maybe then you would be motivated to get me something that I am more suited too.
This is the problem - the gap between the many people with the skills and the actual work has been filled with huge "talent agencies" and programs developed by the likes of myself. Programs that perhaps don't reflect a person's skills or character.
I have HP-UX on my resume, NOT UX. Perhaps I could do this job with proper coaching, but would you actually take the time to read my resume and find out? Give me some Skype time?
[snip]
I realize costs have been cut world wide in a lot of industries but this whole process adds a layer of complexity that I find fraustrating. How many times a day do I need to update my resume with "keywords"? before I actually get an interview with you, then maybe the other two people and then the test?
I've been with your agency for over 2 years, maybe longer. I have applied for:
  • Unix / Linux System Administration (the buzz word is "Engineer" or "Designer" these days)
  • Windows Server Administration 2003, 2008, 2012 - one year relevant experience
  • CISCO CCNA 2 Network Admin Certificate - with 2 1/2 years relevant experience managing a LAN / WAN and its hardware (modems, routers, cabling, switches, etc)
  • Experience with all workstations - 15 years of experience building, troubleshooting, designing, managing.
  • AMD and Intel architecture hardware - 4 years of in depth knowledge that makes me as good as at least a 1st year apprentice electronics technician
  • iphones, android phones - very good knowledge
  • Virtual machines - very good - set up at least 15
  • Scripting in Perl and Shell - very good
  • Programming in Visual Basic and C++ - good
  • Database scripting (MySql) - good
[snip]
Sincerely,
BP

Hi B, 
I Hope you have been well. 
Are you looking for an exciting 3-6 month contract opportunity in the UX design space? 
My client is looking for a UX Designer who will need to work on an old product that needs re-vamping. As the sole UX Designer you will design, lead, and analyze end user needs and leverage the findings into site architectures, Wire-frames, and functional specification documentation for web sites and applications. The UX designer will collaborate closely with a product manager and developers to create the best possible user experience. 
[snip]
If this sounds like you please register your immediate interest by sending through your updated CV and a link to your portfolio/creative work. 
Kind regards,
XXXX
Progressive ICT

Business Names Search
https://connectonline.asic.gov.au/RegistrySearch/faces/landing/bn/SearchBnRegisters.jspx?_adf.ctrl-state=ig8ji303r_13

ORGTEL, TAS BN01642042, Registration Date 01/10/2009
http://www.search.asic.gov.au/cgi-bin/gns030c?acn=139699148&juris=9&hdtext=&srchsrc=1

Name STHREE AUSTRALIA PTY LIMITED
ACN 126 409 103
ABN 86 126 409 103
Registration Date 06/07/2007
http://www.search.asic.gov.au/cgi-bin/gns030c?acn=126409103&juris=9&hdtext=ACN&srchsrc=1

Current details for ABN: 86 126 409 103, ACN: 126 409 103
STHREE AUSTRALIA PTY LIMITED, from 1-Dec-2007
HUXLEY ASSOCIATES, Business Name
http://abn.business.gov.au/SearchByAbn.aspx?StartSearch=True&SearchText=126409103

Name: STHREE AUSTRALIA PTY LIMITED
ACN: 126 409 103
ABN: 86 126 409 103
Registration date: 6/07/2007
Next review date: 6/07/2014
Former name(s): PROGRESSIVE RECRUITMENT PTY LIMITED

NOTE: There are no current entity, trading or business name registered in Australia besides "SThree Australia" and "Huxley Associates". Claims that the other entities are "Trading Divisions" within Australia can only be internal accounting arrangements, not backed by any formal entity or registration.

Only in Tasmania is there a registered trading name: "Orgtel."


Progressive GE (Global Energy)
http://www.progressivege.com/about-us
http://www.progressivege.com/company-details

Progressive Recruitment
http://www.progressiverecruitment.com/about-us
http://www.progressiverecruitment.com/company-details

Huxley Associates
http://www.huxley.com/company-details

Real Staffing
http://www.realstaffing.com/company-details


SThree info.
http://www.sthree.com/about-us/our-philosophy

SThree on Wikipedia
http://en.wikipedia.org/wiki/SThree

SThree History via Archive.org. Started 1986 as "Computer Futures", Taken Public in 2005.
https://web.archive.org/web/20130207013722/http://www.sthree.com/en/page/history/

SThree Investment Case: High Value, High Margin
https://web.archive.org/web/20130207013827/http://www.sthree.com/en/page/investment_case/

Live site
http://www.sthree.com/investors/investment-case

SThree global businesses
http://assets.sthree.com/legal/SThree_Group_Companies.htm


Gary Goldsmith, one of original owners of "Computer Futures". Described as "Colourful.
http://www.express.co.uk/news/royal/378060/Duchess-of-Cambridge-s-very-colourful-uncle

Treffry v SThree Australia Pty Ltd [2013] FWC 3697.
Odd behaviour by both employee and company is specifically commented upon.
http://www.vta.vic.edu.au/docs/News/HR%20Update%20August%202013.pdf
Senior Deputy President (SDP) Boulton found that there was a valid reason for terminaton statng that “there is evidence of dishonesty which was premeditated, preconceived, and which was perpetuated, even after it had been challenged by SThree and that Mr Treffry was warned about possible disciplinary acton”. Further to that, he noted that given the relatvely short period of employment, being about one year and three months, the fact that the conduct of Mr Treffry was clearly dishonest, it would have the potental to undermine the trust and confidence which is necessary in the employment relatonship. SDP Boulton did have some reservatons as to whether the terminaton might be correctly characterised as being one for serious misconduct. “In any event, I am not satsfied that the characterisaton of the terminaton as being for serious misconduct would be such as to outweigh all the other consideratons which have led to the conclusion that the terminaton of Mr Treffry’s employment was not harsh, unjust or unreasonable”.

ACMA, 2012. Formal Warning under Spam Act issued to SThree Australia
http://152.91.62.26/Industry/Marketers/Anti-Spam/Ensuring-you-dont-spam/sydney-businesses-failed-to-comply-with-spam-act-i-acma
http://www.acma.gov.au/webwr/_assets/main/lib410040/sthree_australia-s41_formal_warning-6jan2012.pdf


Progressive Recruitment Specialists Reviews - www.progressive.co.uk
http://www.reviewcentre.com/reviews132341.html
http://www.reviewcentre.com/r132341_2_Progressive_Recruitment_Specia.html



Business Names search - '126409103'
Business Names search - '126 409 103'
Business Names search - 'STHREE AUSTRALIA PTY LIMITED'
Organisations & Business Names search - 'STHREE AUSTRALIA PTY LIMITED'



Business name: ORGTEL
Status: Registered
Registration date: 1/10/2009
Renewal date: 1/10/2015
Address for service of documents: Level 9 1 Market St Sydney NSW 2000
Principal place of business: Level 9 1 Market St Sydney NSW 2000
Holder Name: STHREE AUSTRALIA PTY LIMITED
Holder Type: Body Corporate
ABN: 86 126 409 103

Former identifier: BN01642042
Former State/Territory: TAS



Business name: PROGRESSIVE RECRUITMENT
Status: Cancelled
Registration date: 21/01/2008
Renewal date: 21/01/2011
Cancelled date: 21/01/2011

Former identifier: B2068812U
Former State/Territory: VIC



Business name: COMPUTER FUTURES
Status: Cancelled
Registration date: 10/11/2009
Renewal date: 10/11/2012
Cancelled date: 18/02/2011

Former identifier: BN21413800
Former State/Territory: QLD



Business name: COMPUTER FUTURES
Status: Cancelled
Registration date: 26/10/2009
Renewal date: 26/10/2012
Cancelled date: 11/03/2011

Former identifier: B2229987W
Former State/Territory: VIC



Business name: COMPUTER FUTURES
Status: Cancelled
Registration date: 4/11/2009
Renewal date: 4/11/2012
Cancelled date: 18/02/2011

Former identifier: BN01646217
Former State/Territory: TAS



Business name: COMPUTER FUTURES
Status: Cancelled
Registration date: 27/10/2009
Renewal date: 27/10/2012
Cancelled date: 10/03/2011

Former identifier: BN11447059
Former State/Territory: WA



Business name: COMPUTER FUTURES
Status: Cancelled
Registration date: 5/11/2009
Renewal date: 5/11/2012
Cancelled date: 18/02/2011

Former identifier: BN05058880
Former State/Territory: SA



Business name: COMPUTER FUTURES
Status: Cancelled
Registration date: 28/10/2009
Renewal date: 28/10/2012
Cancelled date: 24/03/2011

Former identifier: F00136100
Former State/Territory: ACT



Business name: COMPUTER FUTURES
Status: Cancelled
Registration date: 27/10/2009
Renewal date: 27/10/2012
Cancelled date: 18/02/2011

Former identifier: BN01134048
Former State/Territory: NT



Business name: COMPUTER FUTURES
Status: Cancelled
Registration date: 26/10/2009
Renewal date: 26/10/2012
Cancelled date: 9/03/2011

Former identifier: BN98463286
Former State/Territory: NSW



Business name: JP GRAY
Status: Cancelled
Registration date: 4/11/2009
Renewal date: 4/11/2012
Cancelled date: 18/02/2011

Former identifier: BN01646229
Former State/Territory: TAS



Business name: JP GRAY
Status: Cancelled
Registration date: 26/10/2009
Renewal date: 26/10/2012
Cancelled date: 11/03/2011

Former identifier: B2229974L
Former State/Territory: VIC



Business name: JP GRAY
Status: Cancelled
Registration date: 27/10/2009
Renewal date: 27/10/2012
Cancelled date: 18/02/2011

Former identifier: BN01134050
Former State/Territory: NT



Business name: JP GRAY
Status: Cancelled
Registration date: 26/10/2009
Renewal date: 26/10/2012
Cancelled date: 9/03/2011

Former identifier: BN98463284
Former State/Territory: NSW



Business name: JP GRAY
Status: Cancelled
Registration date: 28/10/2009
Renewal date: 28/10/2012
Cancelled date: 24/03/2011

Former identifier: F00136099
Former State/Territory: ACT



Business name: JP GRAY
Status: Cancelled
Registration date: 27/11/2009
Renewal date: 27/11/2012
Cancelled date: 18/02/2011

Former identifier: BN21436927
Former State/Territory: QLD



Business name: JP GRAY
Status: Cancelled
Registration date: 5/11/2009
Renewal date: 5/11/2012
Cancelled date: 10/03/2011

Former identifier: BN11453424
Former State/Territory: WA



Business name: JP GRAY
Status: Cancelled
Registration date: 5/11/2009
Renewal date: 5/11/2012
Cancelled date: 18/02/2011

Former identifier: BN05058892
Former State/Territory: SA

2014/06/15

RAID++: Erasures aren't Errors

I'm following the Communications / Transmission theory nomenclature of "Erasures" to mean drive failure ('no signal') and "Errors" to mean incorrectly sent & received symbols.

With 5,000 x 2.5" 2TB drives in a single rack, what hardware problems will we experience?
Worldwide, it's assumed a single vendor might sell 100,000 of these arrays.
  • How many read errors should we expect? "Errors"
  • How many drives will fail in a year? "Erasures"
    • I don't have numbers for other parts like fans, PSU's, boards, connectors, RAM.
    • How many RAID rebuilds?
    • How long will they take?
    • How many dual-failures might we expect?
We can use RAID-1 to set the minimum baseline {cost, performance, data-protection/data-loss} to compare other schemes against. Calculations, here.

2014/06/09

RAID++, HDD's and Ownership

The underpinning of Data Storage, especially for large Enterprises, fundamentally shifted when disk drive capacity increase dropped below 7%/year (40% in 5 years, the design-life of HDD's).

Disk Drives are now consumables, not assets, albeit with a 5-year life. The number of Enterprises with 2,000-5,000 drives in their server room and 10-50,000 drives in their desktop fleet is increasing.

I posit the following following from this:

2014/05/26

RAID and the collapse of IBM's mainframe storage business.

What were the technical and non-technical aspects of the collapse in IBM's DASD (mainframe disk) business from 1990 to 1995, from ~$9B and 80% market share to $1.5B and 35%?

2014/05/25

RAID: Timeline of IBM Disk Storage

A highly selective history of Disk Storage referenced later, focussing on IBM's contributions with some of their competition mentioned. This provides context for previous articles, on Lessons from RAID and RAID++. A good general reference is the Computer History Museum. Minicomputers and the laster Super-minicomputers aren't dealt with, although they created the conditions for the collapse of IBM's mainframe business.


Gordon Bell, while at DEC in 1972, posited "Computer Classes" [PDF, Paper on Bells Law] and how exponential technology growth affects products, markets and eventually companies. Market Disruption from "lower" classes of computers is a feature of computing history. A recent powerpoint by him on the topic.

In the 1970's, IBM dominated the computer (mainframe) industry generating more revenue that all its competitors combined. They were named "The BUNCH": Burroughs, UNIVAC, NCR, Control Data Corporation, and Honeywell.

In the 1960's it was "IBM and the Seven Dwarfs", until RCA and General Electric sold to Sperry and Honeywell respectively.

Some items are included on Software Engineering, an on-going challenge in the Industry.

2014/05/23

RAID++: Lessons from the 1990's market collapse for Large Expensive Disk Drives

Following a previous post, this is an examination of the causes of the collapse in IBM's Mainframe DASD (Direct Access Storage Device) business, from an 80% share of an $11.B market in 1990, to ~25% a decade later. (Source PDF). Notably, IBM's DASD business wasn't the only line to suddenly collapse: the Mainframe business as a whole collapsed, creating a loss across the whole business, the first since 1914.

2014/05/22

RAID++: What invention or change is needed now to take us the next 25 years?

For an industry, digital data storage, that's seen disruptions every 25 years, we're either overdue for one or The Next Revolution has arrived, but nobody's noticed.

What will the Next Big Thing in Storage look like? What, if anything, will succeed RAID, if it hasn't already? Are their lessons we can learn by examining the last big disruption circa 1990 in Storage: RAID arrays?

The one certain lesson from 1955 and 1990 is that nobody can guess, not even remotely, what Data Storage will look like in 25 years time. Any prediction of the 2040 market will be wildly inaccurate, but we can talk about current market forces and technologies and where the trends point for the next 5 and 10 years.

2014/03/30

The New Disruption in Computing

Since 2000, we've been progressively bumping into limits, or end-points, in all areas of silicon technologies. Some I've mentioned previously.

The thesis of this piece is the Next Technology Disruption is No Disruption, No Revolution:
instead of exponential growth of technologies, decreasing unit prices and increasing volume sales, we're now seeing zero or slow growth, steady or increased unit prices (especially if supply-chain is disrupted), and in all but a few market segments, sales are in decline and profits are stressed in many vendors. I believe these are linked.

2014/03/18

Microsoft, IBM and the Price of Success

Microsoft's business is now clearly "under performing" with even the Board admitting it. Steve Ballmer has finally been pushed out. The MSFT Board could've followed IBM in 1993 and appointed an outsider, but they chose an long-term insider.

The imminent "death" of Windows-XP (8-Apr-2014, free security updates end) shows, in my mind, a clear misattribution effect by Microsoft: they've confused cause and effect, that people bought their brand, not just a "standard" commodity product.

2014/01/27

Security: Healthcare, Computers and Ignorance/Inaction.

A year ago I wrote that with the then epidemic of "ransomware" attacks the Hackers had learned how to monetise remote attacks on Healthcare practices. That piece included detailed suggestions on minimum necessary practices and questions for suppliers and vendors.

2014/01/07

Now Read This: Why the Munich Open Source Conversion won't be replicated in Australia.

Whichever side of the Open Source vs Proprietary Software debate you lie on, this article is a "must read". The headline take-away is: "Our goal was 'Freedom', to become independent."
How Munich rejected Steve Ballmer and kicked Microsoft out of the city, Steve Heath, 18th Nov, 2013.

2013/06/25

I.T. is NOT a Profession. There are NO consequences for Failure, It's Unprofessional to not Learn.

A response to a piece on Delimiter reporting QLD and VIC government project failures.

Compare the local IT failures with these comments from Infoq. Author site.
This is best illustrated by the findings from the US Department of Defense (the DoD).[10]
The DoD analysed the results of its software spending, totalling an eye-watering $35.7 billion, during 1995.
They found that only 2 per cent (2%) of the software was able to be used as delivered.
The vast majority, 75 per cent, of the software was either never used or was cancelled prior to delivery.
The remaining 23 per cent of the software was only used following modification.
That would suggest that the DoD actually only received business value from $0.75 billion of its expenditure – nearly $35 billion of its expenditure did not result in software that delivered any immediate business value.
[10] The results of the study were presented at the 5th Annual Joint Aerospace Weapons Systems Support, Sensors, and Simulation Symposium (JAWS S3) in 1999.

2013/06/04

Microsoft: Have missed the boat on catching up to Apple & Google.

In early 2007 I started opining that Microsoft would "hit a financial pothole" around 2010. Thirty posts followed before I gave it up: the mainstream & financial press were looking seriously at the topic. [A Forbes columnist "called the game" in January.]

What prompted my view in late 2006, pre-iPhone, I'd noticed that Microsoft hadn't been able to maintain it's growth above 10%. They (legally) manipulated their figures to remove extreme volatility. Since the early 1970's, I've seen a bunch of Tech companies falter, stumble and fail. Always starting with one "off" result. When the iPhone appeared, I knew the mechanism of their "pothole".

I missed the GFC in 2008 and its effect (revenue down) and in 2012 went on record saying "I missed the date". I'm a technical person, not a financial analyst.

2013/06/02

NBN: Posts moved to a new blog. No more on this blog.

This Blog was intended to be about "I.T." issues in general, but has become overtaken by the NBN.
Posts and Comments have been moved to a dedicated Blog...
Please update any feeds you might have, assuming you want.

I'll be removing the NBN posts from this Blog down the line.