Draft
Filesystem Library Proposal (Revision 3)

This paper proposes a filesystem library component suitable for a C++ Standard Library Technical Specification or for the C++ Standard Library. The proposed library is based on Version 3 of the Boost Filesystem Library (see www.boost.org/libs/filesystem). Preliminary wording is provided. A TODO list identifies remaining work to be done.

Material in this document high-lighted in yellow is known to be incomplete, incorrect, or otherwise require further refinement.

Revision history

D????=12-???? , Filesystem Library Proposal (Revision 3). Changes include:

• Add path::operator+= and concat functions to tack on things like suffixes or numbers. Suggested by Ed Smith-Rowland and others.
• Add section tags and section numbers, per LWG discussion in Kona.
• Bring use of trademarked names into ISO compliance, and minimize use of such names by moving into a single examples section. Thanks to Stefanus Du Toit for advice on how to do this.
• Introduce a definition for "operating system dependent", and then use it to replace most "implementation defined behavior".
• Replace uses of  const string return types with non-const string return types, per LWG discussion in Kona.
• Remove permission for implementations to return const string& in certain cases, per LWG discussion in Kona.
• Remove path inserter and extractor dependency on Boost quoted manip (Issue 7).

N3365=12-0055, Filesystem Library Proposal (Revision 2). Changes include:

• Namespaces changed to std::tbd::filesystem per issue 1 discussion in Kona.
• Removed references to Boost within the proposed wording.
• Minor fixes to the proposed wording.
• Added Issue 7: How should class path inserter and extractor be specified?
• Marked up all issues to reflect Kona LWG discussions.

N3335=12-0025, Filesystem Library for C++11/TR2 (Revision 1). Changes include:

• Regenerated the proposed wording from the Boost Filesystem library reference documentation, using an automated process. This process reduces the likelihood of inadvertent discrepancies between descriptions.
• An Issues list was added, seeded with issues raised by the LWG review of N3239 at the Bloomington meeting, and private communications from LWG members.
• Namespace changed to files as an experiment. Made this issue number 1 so the LWG can pass judgment.
• New functions were added, suggested by David Svoboda, to generate canonical paths and manage permissions.
• More C++11 functionality was applied. This process is still incomplete, however.
• Added proposed changes to header <fstream>. The previous paper had inadvertently supplied the wrong wording.
• Continued the general cleanup of wording.

N3239 = 11-0009, Filesystem Library Update for TR2 (Preliminary), reflected changes made to the Boost library version 3 since the previously accepted committee paper:

• A single class path handles all aspects of internationalization, replacing the previous template and its path and wpath instantiations. Character types char, wchar_t, char16_t, and char32_t are supported. This is a major simplification of the path abstraction, particularly for functions that take path arguments. This change was based on a suggestion by Peter Dimov.
• Several operational functions have been added, including much better symlink support, the ability to resize a file, and the ability to generate a unique path.
• Support for error reporting via error_code is now uniform throughout the operations functions.
• Several functions have been renamed, based on feedback from users.

N1975 = 06-0045, Filesystem Library Proposal for TR2 (Revision 3), was adopted by the committee in April, 2006, at the Berlin meeting. Shortly afterward the Library Working Group set aside work on TR2 to concentrate on C++0x.

Motivation and Scope

The motivation and scope for a filesystem library were described in N1975, and are not repeated here. A minor scope reduction is that an addition to the current C++ runtime library is no longer needed.

Boost Filesystem Version 3 introduced a single path type that interoperates well with both basic_string and user defined string types. Thus the following Design alternatives paragraph is no long applicable:

Single path type which can at runtime accept narrow or wide character pathnames. Although certainly interesting, and possibly superior, such a design would not interoperate well with the current Standard Library's compile-time typed basic_string. A new runtime polymorphic string class would be the best place to experiment with this concept, not a path class.

TODO

• Apply more C++0X features. Boost.Filesystem needs to implement these to verify their application is correct.
• Ion Todirel suggests "Would be nice if path::append was variadic, to improve usability".
• Boost.Filesystem needs to implement char16_t and char32_t support to verify the specification for these is correct.
• The Boost implementation has more class path non-member relational functions that shown in the docs, and the specific set of relational functions varies between Windows and POSIX. Figure out what's happening and document it.
• Source is not specified as actually implemented. Expose path_traits?
• Effects for copy and copy_directory need to be reviewed, revised, tested, peer reviewed.
• Dinkumware/Microsoft report slightly different results for Decomposition table. Rerun table. Check discrepancies.
• Apply issue resolutions from Kona. Complete except some remaining codecvt arguments
• Review Nick Stoughton's email for suggestions, action items.
• Change time_t to chrono system_clock::time_point, per LWG discussion in Kona.
• Ed Smith-Rowland:
  > I found it less confusing to switch to positive logic for recursive_directory_iterator:
  > bool no_push_pending() const noexcept;
  > to say
  > bool is_push_pending() const noexcept;
  >
  > and
  > void no_push(bool value=true);
  > to say
  > void push_pending(bool value=false);
  > or
  > void set_push_pending(bool value=false);
  > I found it less confusing and I think other parts of the library strive for this.
• recursive_directory_iterator is specified in terms of private data members. Remove these (including the comment!) and specify in terms of behavior only.
• Ion Todirel: Would be nice to have a make_relative. Need to review similar suggestions.

Proposed Wording

Gray-shaded italic text is commentary on the proposal. It is not to be added to the working paper.

Filesystem Library [filesystem]

This clause describes components that perform operations on file systems and their components, such as paths, regular files, and directories.

ISO/IEC 9945 contains provisions which, through reference in this text, constitute provisions of this clause. At the time of publication, the editions indicated were valid. All standards are subject to revision, and parties to agreements based on this clause are encouraged to investigate the possibility of applying the most recent editions of the standard indicated below. Members of IEC and ISO maintain registers of currently valid International Standards.

• ISO/IEC 9945:2003, Portable Operating System Interface (POSIX®), part 1 (Base Definitions) and part 2 (System Interfaces), both as corrected by their respective 2004 Correction 1 documents.

  ISO/IEC 9945:2003 is also IEEE Std 1003.1-2001, and The Open Group Base Specifications, Issue 6, and is also known as The Single Unix® Specification, Version 3. It is available from each of those organizations, and may be read online or downloaded from www.unix.org/single_unix_specification/

End of new Clause.

Issues List

Issue 1: What is the appropriate namespace?    Status: Tentatively resolved

Discussion

The N3335 proposal places the library in namespace std::tr2::files. Rationale for a sub-namespace is that the library uses several names that don't seem appropriate for namespace tr2 since full standardization would then put the names into std. The function names remove and rename are of particular concern because these functions differ in behavior from current standard library functions with those names. It also doesn't seem desirable to preempt names like equivalent and status.

Resolution

Kona: Strong support for filesystem as the library's namespace. Strong support for a technical specification namespace that alerts uses that contents are likely to change if and when they later get moved into the standard.

No decision yet on a TR namespace; tbs being used as a placeholder. Thus the full namespace is changed to std::tbs::filesystem.

Issue 2: Excessive use of const codecvt_type& arguments    Status: Open

Discussion

Users sometimes need to do path conversions that use something other than the imbued codecvt facet. The need is particularly acute in multi-threaded applications where changing the imbued facet would introduce a data race. That said, providing an optional const codecvt_type& argument for every function where the need might possibly arise is excessive because its use is so rare and it adds considerable interface clutter.

Proposed resolution

Kona:

Remove all existing class path const codecvt_type& arguments.

Beman to pursue separate encoding conversion functionality, per Thursday N3336 "Adapting standard library strings and IO to a Unicode World" discussion. See Kona wiki.

If a separate encoding conversion facility isn't possible, then some form of minimal class path encoding conversion facility will be needed. But..., the LWG would like to see use cases and clearer explanation of motivation before deciding on specifics, and be very sure that there is no way to achieve with existing std library functionality, albeit with some loss of efficiency.

Issue 3: Possible "implicit cast to native type"?     Status: Open

Discussion

In Bloomington there was discussion of "implicit cast to implicit cast to native OS type to inter operate with existing iostream library and native functions instead of modifying fstream".

Proposed resolution

Kona: The Dinkumware/Microsoft implementation has added an implicit conversion to the string type, eliminating any need to modify fstream. (Discussion: might be safer to make the implicit conversion protected, then make fstream classes friends).

Action: Beman to investigate, test, as avoiding modification of existing header and classes is much desired.

Issue 4: Given move semantics, it is best not to return const strings.     Status: Resolved

Discussion

The issue title pretty much says it all.

Proposed resolution

Kona: Return plain strings. Remove the permission for implementations to return const refs, since uses are not in performace critical code and subtle portability bugs may occur.

Action: Beman to apply to proposed wording.

Version 3: Resolution applied.

Issue 5: Is there a way to handle characters that are illegal for particular OS?     Status: NAD

Discussion

Question raised by Pablo in Bloomington.

Proposed resolution

Beman suggests NAD, Future. I've done some work on this, including looking at systems like OpenVMS that have an escape mechanism to handle otherwise unrepresentable characters. There was a comment to that effect in N3239. I believe it should be deferred to some future release since (1) it is complex enough that I'd like to see actual implementation and use experience (presumably via Boost), and (2) I can't recall a user ever requesting such a feature.

Kona: No consensus for change, since there doesn't appear to be any compelling use case.

Issue 6: Could allocator support be added to class path?     Status: Open

Discussion

Question raised by a committee member in private email.

Kona: Much discussion.

Action: Alisdair to write paper, based on a real implementation, so that it is possible to assess the impact on interface, performance, etc.

Post-Kona: Alisdair will defer paper until after TR.

Issue 7: How should class path inserter and extractor be specified?       Status: Resolved

Discussion

Class path currently is specified to used the Boost detail quoted stream manipulator. A path object is inserted delimited by quotes, and extracted accordingly, ensuring that paths containing spaces are round-tripped correctly.  Quotes in the path itself are escaped, and io state is saved and restored.

See http://www.boost.org/doc/libs/1_48_0/libs/io/doc/quoted_manip.html

Since the standard can't specify behavior in terms of something in Boost, we have to change the specification. Since the quoted stream manipulator is a handy little component, the issue is raised as to whether it should go in TR2.

Resolution

Kona: Specify the path inserter and extractor to have the desired behavior without reference to how it is achieved. (If someone wants to propose a quoted manipulator, that's a separate proposal for a different TR.)

Action:  Beman to apply to proposed wording.

Version 3: Resolution applied.

Issue 8: Rename rename       Status: New

Discussion

There are minor problems with the name of the rename() function:

• There is already a "rename" function, albeit in namespace std, and it has different semantics as regards error handling. Thus giving the function a different name might reduce the chance of user error.
• The name "move" would  better reflect the actual semantics, particularly for moves between directories., and this is the name used by some API's, console commands, and GUI file managers for that functionality.

Proposed resolution

Change the name of the rename() function to move().

$endid $id backmatter= $endid