Latest articles

DAuth - Authentication Utility Lib for D - Initial Release v0.5.1

I've put up an initial release of DAuth: A simple-yet-flexible salted password hash based authentication utility lib for D.

Before you get too excited, know that actual cryptographic algorithms are outside the scope of this lib. Instead, it uses any Phobos-compatible digests and random number generators.

The upside: This makes DAuth fully extensible with plug-in cryptographic algorithms. The downside: It's currently limited to what little exists in Phobos right now (or in any Phobos-compatible third-party digests/RNGs I may not know about).

Full overview, sample code, source and (ugly) API reference are at DAuth's GitHub homepage.

(DUB project name "dauth")

Announcement thread at the D newsgroups is here.

Read more

Google Pulls a Microsoft, Pisses All Over Standards

This one page alone (discovered while searching for why my POP access to Gmail[1] is borderline broken) is sufficient proof that the dumbfucks working at Google have their heads completely up their retard asses, and are hard at work mimicking Microsoft's widely-ridiculed practice of pissing all over standards just for the fuck of pissing all over standards:

So Google expects people to use a non-standard convention in order to make what I'll call "Google POP3" behave like NORMAL FUCK STANDARD POP3?! Except even that still isn't normal POP3 behavior (unconditionally ignores anything beyond 30 days old whether it's been downloaded or not, and retrieves outgoing messages as if they had been incoming).

And yet, the motherfuckers over at Google have the nerve to pretend to be better than Microsoft? Fuck the whole lot of them. For fuck's sake, it's no secret that many of them came from Microsoft in the first place, so it shouldn't surprise anyone that they are the new Microsoft.

[1] "Uhh, if you hate Gmail so much, why do you use it?" Normally I don't, I run my own mail server. But anytime you do that, you still need an administrative contact address that's separate from your own server and domain. Otherwise, if you hit any registrar or hosting snags (and don't think you won't), then your "official" contact address goes right down with the ship, and now you're really in trouble. "Why not Hotmail or Yahoo Mail?" Uhh, would you use Hotmail or Yahoo without anyone pressing a gun to your head?

Read more

Trivially Make Your Site Work on Mobile? Yes, Please!

Want a super easy way to drastically improve your web site's usability on mobile browsers? Quit using CSS overlay popup windows, ie "pop-ins".

It's bad enough that they've always been a terrible practice on desktop browsers (it's a trivial way to guarantee breaking basic browser functionality like back/forward/linking/etc). But on mobile browsers they completely break scrolling and result in pages that are completely unusable. (<-- Follow that link on a mobile browser and then try any of the "Read more" links)

So get rid of CSS overlay popup windows (pop-ins), replace them will full-fledged pages, and your site will instantly improve on desktop browsers and suddenly become usable on mobile browsers.

Read more

Static Checks vs Templates: Always Instantiate Your Templates

One of the key benefits of static typing is the compile-time checking it enables. Compile your program successfully, and entire classes of programmer errors are guaranteed not to exist - regardless of how thorough your unittests are. (Unittests can't realistically be as thorough as static compiler checks. They're an important supplement to static compile-time checks, but not a replacement for them.)

Unfortunately, templates leave a slight hole in this system of guarantees. By the very nature of templates, the code inside cannot be compiled unless the template is actually instantiated. And if the code isn't compiled, it can't be statically checked (beyond mere syntax checks, anyway).

So you're left with this potential scenario: You write a new template in some library, or make some changes/additions to an existing template, then try out your test project and unittest suite. Everything compiles and runs successfully, so you commit your changes and push to the VCS server. Another coder pulls your work, tries to use your template...and gets compiler errors.

Oops! Your test project, or your unittest suite, never actually used the template! Or maybe it only used one instantiation and never attempted other instantiations that were intended to work. Your user became the first one to actually attempt compiling your code, and became the first to discover it didn't work.

Due to the nature of templates, it's impossible for the compiler to test all possible instantiations of a template to ensure they compile (or don't compile, in some cases) as expected, particularly if the template is part of a library. With sufficient sophistication, it may be possible for a compiler to guess at a few sets of arguments that satisfy your template's parameter constraints and then check that they compile. But even D doesn't currently do that.

The moral of this story: Always instantiate your templates. Make sure you always have at least some dummy instantiations that are expected to work. Here's an example of doing it in D:

struct FooType(T) { ... } private alias test_FooType = TypeTuple!( FooType!int, FooType!string, FooType!char ); void fooFunc(T)() { ... } private alias test_fooFunc = TypeTuple!( fooFunc!int, fooFunc!string, fooFunc!(FooType!int) );

It's better, of course, to unittest them. (Tip: It can often help to use foreach over a TypeTuple of the instantiations you want to unittest.) But at the very least, be sure to have some basic instantiations. It's only a one-liner, after all, and may very well save you some embarrassment.

Read more

Scriptlike v0.6.0

An update to Scriptlike is out. Scriptlike is a utility library to aid writing script-like programs in D.

Changes from v0.5.0 to v0.6.0:

  • Change: Path and Ext are now aliases for the UTF-8 instantiations, and the template structs are now named PathT and ExtT.
  • Change: Removed path() and ext() helper functions to free up useful names from the namespace, since they are no longer needed. Use Path() and Ext() instead.
  • Change: Internally split into separate modules, but uses package.d to preserve import scriptlike;.
  • Change: Rename escapeShellPath -> escapeShellArg.
  • Change: Rename runShell -> tryRun. Temporarily keep runShell as an alias.
  • Change: Rename scriptlikeTraceCommands -> scriptlikeEcho. Temporarily keep scriptlikeTraceCommands as an alias.
  • Enhancement: Added scripts to run unittests and build API docs.
  • Enhancement: Added opCast!bool for Path and Ext.
  • Enhancement: fail() no longer requires any boilerplate in main(). NG link
  • Enhancement: Added run to run a shell command like tryRun, but automatically throw if the process returns a non-zero error level.
  • Enhancement: #2: Optional callback sink for command echoing: scriptlikeCustomEcho.
  • Enhancement: #8: Dry run support via bool scriptlikeDryRun.
  • Enhancement: #13: Added ArgsT (and Args helper alias) to safely build command strings from parts.
  • Enhancement: Added this changelog.
  • Fixed: Path(null) and Ext(null) were automatically changed to empty string.
  • Fixed: #10: Docs should include all OS-specific functions.

Read more

Scriptlike: Shell Scripting in D: Annoyances and a Library Solution

Due to its power and general-purpose intent, D has great potential as a faster, easier (in terms of flow-control), and cross-platform alternative to shell scripting. But after quite a bit of experience using it as such, I've noticed several annoyances that consistently slow me down and over-complicate my D scripts:

  • Managing paths as stings is messy and error-prone: There are a lot of fantastic tools in std.path to deal with paths, but there are still annoyances:

    • No static compile-time checks to keep non-path strings out of an API that requires a path.

    • Extra naming conventions are sometimes needed, to help distinguish paths from other stings. Not a major problem, but it's still one more thing to keep consistent, and one more thing making your variable names longer.

    • Extra care required to make sure directories are properly separated by slashes: The buildPath function can help with this, but it's easy to forget and often very tempting to omit.

    • Slashes vs backslashes: Posix uses forward slashes. Windows almost always accepts forward or backslashes interchangeably, but there are a few occasional exceptions. Plus, displaying forward-slashed paths in Windows is bad style.

      D's recommended style is to always use the platform-native slash in all your internal variables, but this can complicate algorithms and make it harder to guarantee your software properly accepts both versions on any OS (which is good style). It also leads to awkward, verbose, easy-to-forget-or-deliberately-stop-caring ugliness like writing "some/big/path" as buildPath("some", "big", "path") or as "some"~dirSeparator~"big"~dirSeparator~"path".

      Or you can use forward-slash-only internally (like I do) and only convert to backslashes as needed. But then you have to obsessively sanitize all path inputs, and many of std.path's outputs. Either way, it's not exactly trivial script-like coding.

    • Trailing slashes? You can ignore the matter of whether trailing slashes exist by relying on the error-prone convention of always using buildPath, even in trivial cases where it verbosifies code and tempts you to omit it (and accept buildPath's unwillingness you let you use "all paths are internally forward-slashed" convention to not complicate your internal logic with "forward and/or back" concerns). Or you can use an "always trailing slash" or "never trailing slash" convention, try to remember which one you chose, and obsessively sanitize all function inputs and external return values while maybe avoiding ugly double-slashes and definitely taking care not to accidentally convert null paths into root directories. Neither options are particularly appealing for shell-like scripts.

  • Invoking a tool that's in the current directory is platform-specific: On Posix, you must prefix `./`. On Windows you must not prefix `./` (but `.\` will work and so will omitting the path). So more buildPath and dirSeparator, just to invoke a tool in the current directory! Whee! Obviously more the fault of the system's command interpreters than D, but still an annoyance that could be improved.

  • Invoking a command shell-style: D's std.process was overhauled a few versions ago and, for the most part, it's fantastic. Easy to capture the child process's output if you want, and any other way you might want to handle the child's standard pipes is fully configurable, as well as invoking the process synchronously or asynchronously.

    But want the standard shell-script style of "synchronously run a shell command and automatically forward all the child's std pipes through the parent's"? You can do it, but for something so common, it's not very obvious how. Examining the documentation, the correct way is supposed to be spawnShell(cmd).wait();. But currently, that fails on Windows when the command's path has spaces, even when the path is properly quoted. Until that's fixed (it's actually fixed in master as of two days ago and should appear in v2.066), the only real solution, AFAICT, is to use the old system(cmd), even though it's "scheduled for deprecation".

  • Proper quoting: std.process has convenient tools for properly quoting commands and arguments. Unfortunately, it's easy to forget to call these functions whenever appropriate, which leads to easy "fails on spaces" bugs. Additionally, on Windows the quoting is done in an oddball way that's rather ugly when echoed (for tracing) and doesn't even work for command names given to system, which as explained above, is necessary for invoking commands shell-style.

  • Command tracing: Optionally echiong all the commands your script runs can be helpful, but it means writing a wrapper to use for all your std.process calls. Not a big problem, but it's one more thing to do and one more thing to clutter your script.

  • Managing imports: Adding proper imports for every part of the standard library I commonly use is perfectly fine for bigger software, but for shell-like scripts it's largely just pedantic boilerplate.

  • Safety when changing the working directory: Often you'll need to run a command in a specific directory. But as soon as you change directories, all your relative paths silently become invalid (They're relative paths, after all). Which means being extra careful to not use the wrong relative paths, and extra work and code to either maintain absolute versions of those paths or make sure everything stays in absolute form (which then results in longer, noisier command lines and tracing output...unless you go to the extra work of sanitizing that too).

    Some of that work and danger can be mitigated by using this idiom every time you change the current working directory:

    { auto saveCwd = getcwd(); chdir("some/dir"); scope(exit) chdir(saveCwd); work... }

    But beyond occasional usage, that gets to be quite a bit of clutter and bookkeeping.

  • Ignoring pedantic filesystem exceptions: Sometimes you just need to make sure a directory exists, creating it if necessary. Or check whether something is a file or directory while counting "doesn't exist" as "duh, no it isn't a file/directory". The exceptions std.path throws in such cases are often a very good sanity check, guaranteeing careless errors don't get through, so I'm glad they exist. But when doing shell-like scripting, they're often more of a bother than they're worth. There are always easy ways to "do it the right way", such as checking whether a path exists before calling isFile or mkdirRecurse, but for script-like purposes these are often unnecessary concerns and only add clutter to otherwise straightforward code. Wrappers are easy to make, but that's still one more thing to do, and that much more boilerplate for your scripts.

  • Deleting an entire directory tree: Normally as easy as rm foo -rf (if you don't care about Windows compatibility), there's no similarly simple equivalent in std.file.

  • Copy and rename aren't as powerful: In a shell script, copy and move can optionally take a glob or directory as the source (instead of just a file), and a directory as the destination. But std.file's equivalents can only take individual files as both source and destination. And std.file's remove and rmdir don't accept globs either.

  • Remove isn't as powerful: Like copy and rename, remove and rmdir don't accept globs. And unlike Posix's rm, a single command can't be used to delete either files or directories. To delete a path generically, you need to first check if it's a file or a directory and call the appropriate function.

  • Accessing the filesystem with UTF-16 or UTF-32 paths: The functions in std.file only accept paths and file names in UTF-8. Although it hasn't been an issue for me personally, if you do have paths in UTF-16 or UTF-32, they need to be converted manually with to!string() or text(). This is a reasonable default since it reduces hidden allocations, but for script-like uses it could be a nuisance.

Some of these things could, and probably should, be addressed in D's standard library, Phobos. But many other issues are arguably less appropriate for Phobos for various reasons. To address all of these issues, and perhaps others that may surface, I've created a simple one-file library, Scriptlike.

At the moment, not all of the issues above are addressed by Scriptlike just yet. But the rest is ready-to-go, and you can check the issue tracker to see what work still remains. Scriptlike is also available in DUB as package scriptlike.

Here are the current features in Scriptlike:

  • A thin wrapper over std.path and std.file that provides a dedicated Path type specifically designed for managing file paths in a simple, reliable, cross-platform way. No more dealing with slashes, paths-with-spaces, calling buildPath, normalizing, or getting paths mixed up with ordinary strings.
  • Optionally enable automatic command echoing (including shell commands, changing/creating directories and deleting/copying/moving/linking/renaming both directories and files) by setting one simple flag: bool scriptlikeTraceCommands
  • Most typical Phobos modules automatically imported. Who needs rows and rows of standard lib imports for a mere script?
  • Less-pedantic filesystem operations for when you don't care whether it exists or not: existsAsFile, existsAsDir, existsAsSymlink, tryRename, trySymlink, tryCopy, tryMkdir, tryMkdirRecurse, tryRmdir, tryRmdirRecurse, tryRemove: All check whether the source path exists and return WITHOUT throwing if there's nothing to do.
  • One simple call, runShell, to run a shell command script-style (ie, synchronously with forwarded stdout/in/err) from any working directory. (Also automatically works around DMD #10863 without waiting for v2.066.)
  • One simple function, fail(string msg), to help you exit with an error message in an exception-safe way. (Does require some minor boilerplate added to your main().)
  • More to come!

Read more