Monthly Archives: October 2012

Moving to WordPress – Part 1: Content

As I mentioned in a previous post, I am done with Movable Type, and I’m in the process of moving this blog to WordPress. That turns out to be a lot easier said than done.

(Warning: Much technical computer geekery ahead.)

WordPress has a Movable Type import tool, but It doesn’t solve what I consider to be the most important problem: It doesn’t preserve permalinks. Over the years, many people have linked to posts on Windypundit — Google webmaster tools reports over 90,000 inbound links — and I want to make sure that as many as possible continue to work. Not only is it basic ettiquette not to break links, but having working links to my site is also important to maintaining what passes for my search engine rankings.

I searched for other tools and methods but I couldn’t find anything that would solve the permalinks problem without introducing other problems of some kind. Eventually, out of frustration and a desire to learn new tricks, I decided on a rather crazy course of action: I decided to port my blog to WordPress by writing my own program.

I chose to write it in C#, mostly because I’d just been hired for a job programming in C# and I figured writing the importer would be a good way to learn C#. Also, I already owned Microsoft Visual Studio 2010 Professional and I’m comfortable with the development environment.

That was a year ago. I got in enough C# practice that I had no trouble when I started the job, but I kind of lost interest in writing the blog migration tool. Development slowed to a crawl.

A few weeks ago, however, I started getting a lot of spam comments that weren’t being caught by my anti-spam tools. Knowing that I’d have much better tools if I switched to WordPress, I decided it was in my best interest to get the job done.

The main tool I wrote is the BlogMigrator, which pulls all of my posts out of Movable Type and generates a WordPress extended RSS (WXR) file suitable for import into WordPress.

It starts by downloading all the posts from the current Windypundit website, which it does by connecting directly to the MySql database that Movable Type uses to store all my posts. It queries to get all the authors, categories, posts, and comments in the blog, including all pages (such as the About pages) and all draft posts loaded into an ADO.NET dataset, which it saves to a file. From then on, the migrator just loads blog posts from the file, avoiding the time consuming download from the server. If I want the latest stuff instead of my locally cached copy, I just delete the DataSet file and the BlogMigrator downloads a new one next time I run it.

Next the program builds an in-memory model of the Movable Type blog, including all authories, categories, blog entries, template maps, and comments. This is a fairly mechanical process. After that’s done, it iterates over all the blog entries and generates a report of the location and publishing status of every post. I use these reports (and others) as input to the iterative development process.

The next step is to traverse the Movable Type blog model and build a matching WordPress blog model. All of the basic concepts are the same, but there are a lot of little details that change, including the names of the data fields, and I try to follow the naming conventions of each blog technology as much as I can. (E.g. Movable Type author have single name field, WordPress authors have first and last names.) Among the steps of the conversion are splitting the author name into first and last names, generating unique IDs for each post, merging main and extended post text, converting the URL format for the post, and converting from local time to universal time.

The next pass looks into the actual content of each post and uses the HTML Agility Pack to analyze the HTML and catalog every element and all the class and style attributes. It also generates reports of which posts each of those items is used in. I’ve been using those reports to make iterative modifications to the program. For example, some posts have embedded HTML class junk that was introduced when blog authors cut-and-pasted from Microsoft Word. By finding these classes in the reports, I was able to modify the program to strip them out. I have a whole collection of whitelists, blacklists, and replacement tables.

In other cases, where the reports showed that a strange class or misspelled element was used in only a handful of posts, I’ve just gone back in to the blog on MovableType to fix the problem at the source, which is easier than adding code to fix the problem. Then I re-import the database and re-run the BlogMigrator to confirm the problem is fixed. (For problems fixed by the tool, there are before- and after-cleanup reports so I can verify the problem is gone.)

Another thing I had to handle was custom tags. MovableType allows you to create custom HTML-like tags for your blog by writing a little PHP and/or Perl code, and I had built a few of them over the years. I had to modify the HTML Agility Pack to recognize them as legitimate tags, and then I had the BlogMigrator replace them either by generating raw HTML or by re-writing them into custom WordPress shortcodes (which are a similar concept to MT’s custom tags). Fortunately, I was mostly able to implement the shortcodes by reusing the PHP code I had already written.

The BlogMigrator also catalogs all the links and images in each post, generating CSV files that I can view in Excel. Each link is identified as internal — back to Windypundit — or external depending on the hostname. Internal links are further classified by checking whether they point to a known Windypundit post URL or something else, such as an image or sound file.

I then have a separate AssetDownloader program that reads in these report files and downloads all the assets on the site and builds a directory structure for them. It filters out file types that are not static assets, such as links to .php files. I can upload all the files in that directory to the new website so the internal links work, although the program rewrites them with a new top-level subdirectory so they won’t collide with the new blog’s native assets. It also cleans up problems like replacing spaces in the URLs with underscores.

I then have a third program, the Probulator, that reads the link report, rewrites every link to point at the new blog, and tests the link to make sure it works. The first time, it found 61 broken internal links, due to basename shortening, badly formed URLs, embedded spaces, and so on. I went back to the fix (or remove) the links.

It also found a couple of dozen links that were broken because my program was using the date of publication of a post to generate the URL and Movable Type was using the date of creation (or vice versa, I can’t remember).

The Probulator also tries to download a copy of every blog post by using its original URL — except for the hostname, which it rewrites to refer to my test server. This serves as a test for broken links, and it also provides local copies for further analysis by two more programs.

The SanityChecker program examines each blog post for odd bits of HTML that might not format correctly. For example, all post content should be a list of a limited set of tags — <p>, <ul>, <ol>, <h5>, <h6> — or <blockquote>, which should contain a list of the same set of tags. The program reported anything that did not match that pattern. This uncovered a flaw in my implementation of one of the shortcode replacements for custom tags. It also found a bunch of posts which had been authored using mangled HTML. (God bless Joel Rosenberg’s memory, but he didn’t know a damned thing about HTML.) I had to go back to the Movable Type version of the blog to clean those up before importing.

The last program is the LinkVerifier, which finds all the links in every post and makes sure that all the internal ones still work. A few of the links are to blog-engine-specific resources, such as category archives and author about pages, that can’t be easily mapped. I’ll keep a list of those so I can go in and fix them later.

At this point, my process for a full import goes something like this:

  • Delete the database cache file (if I’ve changed something on the Windypundit site and I want to re-download everything).
  • Run the BlogMigrator.
  • Run the AssetDownloader.
  • Zip up the downloaded assets, upload them to the new blog host, and extract them into the proper directory.
  • On the new blog host, restore a backup of the WordPress database that has all the configuration items set but doesn’t have any posts in it.
  • Use the WordPress importer to upload the WXR file from the BlogMigrator that contains all the posts.
  • Hit the blog homepage to verify that it’s working.
  • Run the Probulator, check the reports for missing items.
  • Run the SanityChecker, check the reports for problems.
  • Run the LinkVerifier, check the reports for problems.
  • Go fix some problems and try the process again.

I’ve pretty much been doing that in my spare time for the past couple of weeks, and I think I’m almost done. I’ll probably roll out the live site in the next few days.

Zumba, Libertarian Style

I’m a regular reader of Jack Marshall’s Ethics Alarms blog, in which he discusses ethical issues that arise in the news of the day. He calls out a lot of public figures for their shady ethics, but he also gives them credit when they get something right. I agree with much of what he has to say, but when his ethical analysis crosses over into matters of public policy — crime, immigration, and the free market — his analysis is not friendly to libertarian values, and I often find myself in angry disagreement.

A few days ago he wrote about the Zumba fitness instructor in Kennebunk, Maine who has been charged with prostitution. The cops claim to have her client list and detailed records, and as they investigated those men, they apparently planned to release their names.

Jack concludes, rightly, that it’s not a good idea to release the names, but in the course of the analysis, he gets a lot wrong.

We begin, on Ethics Alarms, at least, with the premise that prostitution is damaging to the fabric of a healthy society, and should be prohibited by law. I’m not going to debate that issue again here, not today. My position is that prostitution is not a “victimless crime,” that it exploits and traps women and destroys families and lives, and government has a legitimate interest in forbidding it. I know the libertarians disagree. That’s not the issue here.

Fair enough. He’s interested in a particular issue, and he doesn’t want to talk something he considers out-of-scope. I didn’t leave any comments there. But I’m sure as hell going to talk about it here.

My position is that prostitution is a consensual crime — a crime that makes it illegal to do something even though everyone involved has given their competent consent — and that it is absurd and immoral to imprison people for such crimes. It is prison that traps people, and it is prison that destroys families and lives. Government has no legitimate reason for interfering with consensual behavior. That is the issue here.

When you’ve been a libertarian — or classical liberal or whatever you call it — for as long as I have, debates over things like whether the cops should release the client lists begin to seem very strange. It’s like listening to a debate about a black man marrying a white woman in which both sides want to throw the black guy in prison, and the debate is whether or not the white woman should be shunned by her family. There’s a difference of opinion, and I know which side I’d pick (if those were the only two), but it really seems to miss the point in a bad way.

Anyway, Jack presents his analysis:

That brings us to ethical balancing, or Utilitarianism. There are a set of predictable desirable and undesirable results that will come from releasing the names, and another set that will result from not releasing them:

Since I accept the libertarian position, my utilitarian calculation is a little different from Jack’s…

Release the names: Pro.

It’s fair, since woman involved already has had her identity made public.

Nope. It was none of the cops’ business when they released her name, and it’s still none of the cops’ business. Two wrongs don’t make a right.

The shaming will help enforce the law, and discourage violators.

The law has no legitmate purpose, so nothing good comes from enforcing it or encouraging people to obey it.

The violators have no right to keep their sexual infidelity from spouses and lovers. They are accountable and responsible for their actions, and this is a completely predictable and reasonable consequence that they knowingly risked occurring.

That something is predictable doesn’t make it right. If they walked around in a bad neighborhood late at night carrying a pile of cash in a see-through plastic baggy, it’s highly likely they would get mugged. This too is a completely predictable consequence that they knowingly risked occurring, but that doesn’t mean the mugger is in the right.

Similarly, just because they could have predicted the possibility of shame doesn’t make it right for the police to cause that shame.

Public officials particularly should be exposed when they break laws and defy community standards of conduct.

Agreed in general. But it doesn’t help in this case. 

If family members are adversely affected, that is not the proper concern of the law, the courts or society. Whatever harm they suffer is the result of the lawbreaker’s anti-social conduct.

No, no, a thousand times no!

One of the recurring themes of this blog is that when doing a utilitarian calculation, all the costs and benefits matter. Criminals are harmed by the justice system, many of them quite severely, locked up under dreadful conditions. A proper utilitarian calculation doesn’t pretend the criminals don’t suffer, rather it acknowledges the full measure of their suffering and then examines whether that suffering is necessary to prevent even greater suffering.

Sure, when the clients decided to hire a prostitute, they should have taken into account the possibility of discovery and the subsequent harm to their families if their behavior is publicized. Their decisions may very well make them bad people. But the question now before us is not whether they did the wrong thing, but how we (or the police acting in our names) should respond if we want to do the right thing.

If the police reveal the clients’ names, it will almost certainly hurt their family members. The utilitarian question is whether revealing the names will produce benefits, fully accounted for, that will be worth the cost to the families.

Seeing little benefit, I vote no.

If society protects wrongdoers from the natural consequences of their conduct, it encourages wrongdoing.

Agreed completely. Of course in my libertarian view, prostituion by itself is not wrongdoing. Rather, it is the cops and the prosecutors who are the wrongdoers, butting in where they have no right, interfering with the free market transactions of consenting men and women.

I pretty much agree with all the cons:

Release the names: Con

The harm is vastly disproportionate to the conduct.

Those whose names are revealed will lose jobs; families will be disrupted, careers and reputations will be destroyed.

There will be no degrees of shame. One time clients will be as humiliated as regulars.

The presence of the name on the list is insufficient evidence, on its own, to justify the degree of harm that will result from the list’s release. They will be pronounced guilty without trials or due process.

Making such lists public will encourage prostitutes to seed their clients lists with the names of prominent individuals as leverage in the event of their arrest.

A few more notes:

Several news stories suggest that the alleged prostitute may have been working with a private investigator to record her sex acts with clients and then possibly blackmail them. If that’s true, what she did is in no way a consensual crime. It’s much worse than the prostitution.

Since I started writing this, the police have revealed a batch of names although I haven’t seen any of them in the news yet. Maybe it wasn’t anyone newsworthy.

The men have apparently been charged with a “Class E misdemeanor of engaging a prostitute,” which makes the release of names somewhat legitimate, since we (mostly) don’t have secret prosecutions in this country.

I’m not sure what the evidence is against them, but if it’s only the prostitute’s records, I’m curious how the police will establish what those records mean. Did she confess? Will they immunize her to get her testimony? Do they have some sort of smoke-and-mirrors “prostitution expert” to explain it? Is it all just a scheme to get some of the men to confess?

Sigh. No good will come of this.

Koch Derangement Syndrome at In These Times

(That title is a bit dog-bites-man, isn’t it?)

I’m amused, puzzled, and a bit angered by the stupidity of lefty/liberal/progressive hatred for all things Koch. Mike Elk’s ridiculous piece at In These Times is a particularly confused example:

Much has been written about the owners of Koch Industries, brothers David H. Koch and Charles G. Koch, trying to control the political process through hundreds of millions of dollars in donations to right-wing causes and candidates. Now, an In These Times investigation reveals that the billionaires have broken out another tactic to influence the 2012 elections: attempting to control their workers’ votes.

Controlling their workers votes? That sounds bad. Illegal even. How are they doing that?

In a voter information packet obtained by In These Times, the Koch Industries corporate leadership informed tens of thousands of employees at its subsidiary, Georgia Pacific, that their livelihood could depend on the 2012 election and that the company supports Mitt Romney for president.

Elk goes on to quote from the mailing:

If we elect candidates who want to spend hundreds of billions in borrowed money on costly new subsidies for a few favored cronies, put unprecedented regulatory burdens on businesses, prevent or delay important new construction projects, and excessively hinder free trade, then many of our more than 50,000 U.S. employees and contractors may suffer the consequences, including higher gasoline prices, runaway inflation, and other ills.

Enclosed with the letter was a flyer listing Koch-endorsed candidates, beginning with Romney. Robertson’s letter explained: “At the request of many employees, we have also provided a list of candidates in your state that have been supported by Koch companies or by KOCHPAC, our employee political action committee.”

The packet also included an anti-Obama editorial by Charles Koch and a pro-Romney editorial by David Koch. The letter went on to say, “We believe any decision about which candidates to support is–as always–yours and yours alone, based on the factors that are most important to you. Second, we do not support candidates based on their political affiliation.”

In other words those dastardly Koch brothers are trying to “control their workers’ votes” by talking to them. In this case, by talking to them about political issues — subsidies, regulations, disruption of free trade — that may directly hurt Koch Industries and therefore put employees out of work. The bastards! How dare they keep employees informed about the business environment!

The Koch’s in-house campaigning for the GOP is part of a larger trend of corporations exercising new freedoms under Citizens United. The Supreme Court decision overturned previous FEC laws prohibiting employers from expressing electoral opinions directly to their employees.

This is what the liberal reaction to the Citizens United ruling comes down to: Fear that business leaders will be able to talk about politics with their employees.

(Note the all-too-typical hypocrisy: In These Times is a progressive magazine that exists entirely to express political opinions. You can be damned sure that its employees know their bosses’ politics.)

Elk isn’t just concerned about what the Koch’s are saying, he also accuses them of trying to suppress their employees’ freedom of speech.

In September, a number of unionized employees at Georgia Pacific’s Toledo, Ore. plant posed for a photo in front of their union hall with Democratic state Senate candidate Arnie Roblan. When the Koch Industries voter information packet arrived in the workers’ mailboxes a few weeks later, they saw that Roblan was not on the list of Koch-endorsed candidates in Oregon.

It was then, says Association of Western Pulp and Paper Workers (AWPPW) Vice President Greg Pallesen, that he started receiving some of the strangest phone calls from workers he’s fielded in his 30-plus years of union involvement. The unionized workers in the photo were worried that they might be fired from their jobs if the image got out on the Internet, because in the backdrop of the photo, the Georgia Pacific plant could be seen.

And? And? …Well, nothing, really. If anything bad ever happened to those workers, Elk doesn’t mention it. But, you know, they were worried, so therefore the Koch brothers are evil.

Georgia Pacific workers say that in general, they are not sure where the boundaries of the social media policy lie. AWWPW Local 5 President Jim Pierce, who works at Georgia Pacific paper mill, in Camas, Washington, is wary of commenting online about the outspoken Koch Brothers’ political beliefs.

“Even if I was at my own home, I can’t put something up [on Facebook] against the Koch Brothers,” says Pierce. “I don’t post anything about the Koch Brothers. I could lose my job.”

So, he’s in fear for his job…and yet he feels free to say things like this to reporters. And as far as Elk tells us, nothing happened to him either.

Social media policies are a modern extension of traditional policies controlling who speaks on the company’s behalf. Companies can get in a lot of trouble if random employees start making promises to customers that they’re not authorized to make, or making statements to the press about sensitive internal matters. Companies also don’t want employees doing things that conflict with the company’s business goals and the purpose of the employee’s job. If you sell cars for Lexus by day, you shouldn’t be badmouthing Lexus cars by night on your Facebook page, and you especially shouldn’t be calling your customers idiots for buying them. That’s why employee handbooks include stuff like this.

(This gets more complicated when the issue is one of public policy — can BP employees complain about global warming? — and it can be downright illegal when companies try to keep employees from talking about working conditions, because that would interfere withthe ability of workers to organize and form unions. Elk tells us that the union has two NLRB complaints against the company’s social media policy, so it’s not implausible that that policy goes too far, some of them do. Koch industries could well be doing something wrong if they are trying to suppress such speech.)

Elk also mentions a possible case of retaliation that sounds plausible:

When McKinney applied for a foreman job at the plant in May, he says, his supervisor informed him that a higher-up said he wouldn’t get the job because he was “too political.” “They said I should be aware of what I am posting online,” says McKinney. A subsequent August evaluation of McKinney noted that “supervisors feel Travis gets caught up in the politics of the day which can be distraction.”

McKinney says it wasn’t hard to deduce what they meant. He was quoted in the 2011 Nation article I wrote with Mark Ames…

That doesn’t sound very nice. Then again, if you complain to the news media about your employer’s policies, perhaps they are justified in not giving you a job where you would have to implement those policies.

So far, this is all just standard-issue progressive Koch-hate, possibly with some justification if there has been retaliation over labor issues. But then Elk veers off into deranged political hackery:

In addition to the social media policy, Georgia Pacific also demands that workers seek approval from supervisors before running for local elected office or serving on the boards of nonprofits. Koch Industries claims such approval is necessary to prevent conflicts of interest. These policies could potentially prohibit Georgia Pacific employees from running for local office in communities that seek to more strictly regulate the company.

“I was kind of disturbed that they would infringe on my personal right to run for office,” says Georgia Pacific employee Larry Wagoner of Washougal, Wash. ” I was in the running for City Council this year. I asked someone in the HR department, ‘What if I wanted to run for Congress?’ She said you would just have to stop working here.” Wagoner adds that he is pretty sure this was a misinterpretation of company policy. But it serves as an example of the fuzzy boundaries of the policies and their potential chilling effect.

Provisions like this are also pretty common in employment policies, for exactly the reason given: Conflict of interest. Is Koch Industries supposed to have people working for it that are also holding offices where they can work against it? This is not just about Koch protecting its interests, it’s also about obeying laws about campaign finance and public corruption and avoiding the appearance of impropriety.

You can see how crazy this is. Elk started his article by complaining that the evil Koch brothers were having too much influence over the political process, but he ends it by complaining that Koch Industries doesn’t make it easy enough for its employees should run for elected offices. Obviously, Elk is only imagining non-management employees running for office, people who he assumes are good progressive pro-labor folks like himself. If some Koch VP got elected, people like Elk would be the first ones screaming that he’s a stooge for the Koch brothers.

Nobody’s Firing Big Bird

I get a lot of political email these days, especially from the Obama campaign. Yesterday, for example, I got one from Stephanie Cutter, one of Obama’s campaign managers.

Over the course of this election, we’ve seen that both Romney and Ryan avoid telling the truth about their plans and how they’d actually affect the middle class. It doesn’t matter if they’re talking about taxes, health care, Medicare, education, or clean energy — the Romney-Ryan status quo is to misrepresent their positions and their practical effects.

But Stephanie isn’t above telling a few lies herself:

One of the few specific policy proposals that Romney offered at the debate was to fire Big Bird to cut the deficit. You couldn’t make this stuff up.

Except she kinda did.

PBS only receives about 15% of its funding from the government, so eliminating their federal funding would not be much of a hardship. They might have to cut some programming, but surely not Sesame Street, their most well-known show. And even if they did such a crazy thing, PBS doesn’t own Sesame Street. Big Bird’s employer is actually the Sesame Workshop, a non-profit corporation that makes a lot of money through licensing of its famous characters all over the world. If one of the television networks that carries their video product lost a bit of its funding, they’d still be able to get their product out. Nobody’s firing Big Bird.

On the other hand, I don’t understand why Republicans have a bug up their ass about PBS. I agree that in an age of cable television and the Internet, we really don’t need publicly-subsidized television, but we’re talking about a very small amount of money. If Romney were serious about reducing government spending, he be talking about cutting energy and agricultural subsidies.

The FBI Questions a Stormwatcher

Michael Galindo has a hobby taking pictures of storms as a volunteer for the National Weather Service’s Skywarn program. On September 13 he took a few pictures of a brewing storm which, unknown to him, happened to be near the Lyondell Refinery outside Houston. And I wouldn’t be mentioning this if it hadn’t taken an ugly turn:

A man who snapped photos of a brewing storm last month received a visit Friday from an FBI Agent, inquiring why he would want to take such photos.

Michael Galindo explained that he was simply volunteering for the National Weather Service.

And FBI Agent David Pileggi seemed to be satisfied with that response.

But Galindo was left wondering whether he now has a permanent FBI file.

“He told me, ‘you’re not a threat and you are doing a public service but just be careful next time,'” Galindo said in a telephone interview with Photography is Not a Crime.

With all due respect to Agent Pileggi — who seems to have handled this in a reasonable and proportionate matter — Michael Galindo was just a guy taking some pictures in a public place. That’s his right. You are the the one who got sent on a fool’s errand to harass an American citizen by demanding an explanation for something that he shouldn’t have to explain to anyone.

Next time, maybe you should be more careful.