bluespec.com Forum Index bluespec.com
Bluespec Forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Segfault with large PipelineFIFO vector

 
Post new topic   Reply to topic    bluespec.com Forum Index -> Tools: BSC (Bluespec Compiler)
View previous topic :: View next topic  
Author Message
tmarkettos



Joined: 18 Sep 2012
Posts: 2

PostPosted: Tue Sep 18, 2012 1:19 pm    Post subject: Segfault with large PipelineFIFO vector Reply with quote

I have a large FIFO - 64 stages, where each stage is a 64-vector of 32 bit elements. Total size 128Kbit. See the attached file.

If I use mkFIFO, it compiles and simulates fine but each stage takes two clock cycles (ie a bubble after each stage). Each compile run takes a few minutes (Xeon E5-2680 0 @ 2.70GHz). If I use mkPipelineFIFO or mkSizedFIFO(1), the Bluespec compiler segfaults. If I use mkLFIFO it compiles/simulates fine and each stage takes one cycle.

Code:
$ bsc -print-expiration
Bluespec Compiler, version 2012.07.beta1 (build 29243, 2012-07-26)
$ bsc -keep-fires -cross-info -steps 100000000 -sim -show-schedule -g mkTBMyCore -u TBMyCore.bsv
checking package dependencies
compiling TBMyCore.bsv
code generation for mkTBMyCore starts
Segmentation fault (core dumped)


While working on this code I've also had:
Code:
bsc: internal error: evacuate: strange closure type 67330704
    (GHC version 6.12.3 for x86_64_unknown_linux)
    Please report this as a GHC bug: 
make: *** [TBMyCore.bo] Aborted (core dumped)


So I have a sufficient workaround, but it probably shouldn't segfault with those types.



MyCore.bsv
 Description:
128Kbit pipeline example

Download
 Filename:  MyCore.bsv
 Filesize:  1.77 KB
 Downloaded:  1338 Time(s)

Back to top
View user's profile Send private message
quark
Site Admin


Joined: 02 Nov 2007
Posts: 496

PostPosted: Tue Sep 18, 2012 2:53 pm    Post subject: Re: Segfault with large PipelineFIFO vector Reply with quote

Thank you for reporting this. Both of these behaviors are possible if the BSC process runs out of memory. I ran this example using 2012.07.beta1 and confirmed that the scheduling stage is using too much memory. (If you compile with the "-v" flag, you will see that the segfault occurs during the scheduling stage.)

This problem has since been fixed, so you will see an improvement in the next release -- the example compiles in around 40 seconds now. The fix is available in a more recent beta release, if you would like to download it: http://www.bluespec.com/downloads/Bluespec-2012.09.beta1B.tar.gz

This beta release cuts out unnecessary analysis during the scheduling phase and instead of using BDD analysis, it uses an SMT solver, which requires less memory and is faster.

The difference between "mkPipelineFIFO" and the other FIFOs is that "mkPipelineFIFO" is not separately synthesized. As a result, the rules and logic inside the instantiations of this module become part of the full design, increasing the complexity. The increased number of rules was a problem for the old scheduling analysis. So one way to avoid the problem would be to separately synthesize the pipeline FIFOs, to firewall their internals from the rest of the design:
Code:
(* synthesize *)
module mkPipelineFIFO_V64_B32 (FIFO#(Vector#(64,Bit#(32))));
   let f <- mkPipelineFIFO();
   return f;
endmodule

I confirmed that making this change caused your example to compile in 50 seconds.

You said that using "mkSizedFIFO(1)" also caused a problem, but I am not able to reproduce that. It compiles the same as "mkFIFO" for me. I expect this, because "mkSizedFIFO" is separately synthesized. Only "mkPipelineFIFO" is not.

Note that you can also add a synthesis boundary to the module "mkMyCore", to firewall its internals from the testbench module. Although that does not appear to improve the compile time in this case.

In any case, with a newer release of BSC, you won't have to play these kind of synthesis-boundary games as often.
Back to top
View user's profile Send private message
tmarkettos



Joined: 18 Sep 2012
Posts: 2

PostPosted: Tue Sep 18, 2012 3:09 pm    Post subject: Reply with quote

Thanks for the reply. Looking at 'top', the process takes about 500MB of RAM (VIRT and RES) when it dies, but it does start climbing rapidly just before. This machine has 256GB of RAM and I regularly have processes taking 5GB or so, so memory isn't really a problem. I ran it in gdb but there were no function names to tell me anything interesting.

My apologies, mkSizedFIFO(1) works but it's mkBypassFIFO that also segfaults.

I'll take a look at the new beta, thanks for your help.
Back to top
View user's profile Send private message
quark
Site Admin


Joined: 02 Nov 2007
Posts: 496

PostPosted: Mon Sep 24, 2012 1:46 pm    Post subject: Reply with quote

It turns out that BSC was not exceeding the memory, but that the BDD analysis was running out of free variables to use. This is a limitation of BDD analysis and it happens when the expressions (to be represented as BDDs) get too large. The new SMT analysis does not have this limitation.

Interestingly, the segfault only occurs on some computers. On other computers, BSC issues an error (saying that the BDD analysis ran out of variables). I don't understand why the error is successfully reporting on some systems and not others. But in any case, this won't come up with new releases.
Back to top
View user's profile Send private message
lsrosa



Joined: 20 Jul 2012
Posts: 3

PostPosted: Mon Mar 04, 2013 9:39 pm    Post subject: same problem here Reply with quote

I`m trying to create a really big matrix*matrix architecture (100*100 = 10000 processing elements) ..

I`m using -keep-fires -steps-max-intervals 100 and +RTS -k50M -RTS

but at the 39th interval I get a segmentation fault

how can I prevent this?
Back to top
View user's profile Send private message
quark
Site Admin


Joined: 02 Nov 2007
Posts: 496

PostPosted: Mon Mar 04, 2013 10:26 pm    Post subject: Re: same problem here Reply with quote

If it's the same issue, then it is fixed in the latest compiler. Try using a recent beta release, such as http://bluespec.com/downloads/Bluespec-2013.01.beta7.tar.gz
Back to top
View user's profile Send private message
lsrosa



Joined: 20 Jul 2012
Posts: 3

PostPosted: Sun Aug 04, 2013 8:47 pm    Post subject: Reply with quote

it`s been a year and I've formatted my computer, can I have an updated link to 2013 beta release?[/url]
Back to top
View user's profile Send private message
quark
Site Admin


Joined: 02 Nov 2007
Posts: 496

PostPosted: Sun Aug 04, 2013 9:00 pm    Post subject: Reply with quote

There was an issue with that release, so that URL no longer works. Here is a more recent beta release, which you can use:
http://www.bluespec.com/downloads/Bluespec-2013.05.beta2.tar.gz
Back to top
View user's profile Send private message
ted



Joined: 28 May 2013
Posts: 3

PostPosted: Thu Mar 13, 2014 1:27 pm    Post subject: make: *** [Mult0.bo] Segmentation fault Reply with quote

Hello,

I've downloaded the /Bluespec-2013.05.beta2.gz file and installed
it on a 64-bit Scientific Linux system. I am attempting to run the
"mult0" example found in the .../training/BSV/labs/mult0 install directory.
When I run "make run" the system reports a "Segmentation Fault":

[email protected] mult0 12:29pm >make run
bsc -keep-fires -cross-info -sim -g mkMult0 -u Mult0.bsv
make: *** [Mult0.bo] Segmentation fault
[email protected] mult0 12:29pm >pwd

From the various posts I've read on this forum, other users have
encountered Segmentation Fault errors and apparently the cause of
the problem was to have been fixed in the Bluespec-2013.05.beta2.gz
version.

I am running the bluespec compiler on a 64 bit Scientific Linux 6 system:
[email protected] mult0 12:50pm >uname -r
2.6.32-431.5.1.el6.x86_64


My earlier version of bluespec_2012.01.A worked fine under Scientific Linux 5.

Can someone please tell me what the cause of the error is and whether there is a more recent version of the Bluespec tools for Linux which supports Scientific Linux 6. We are almost halfway through 2014 so a
2013 release is a bit old.

Thank you.

Ted
Back to top
View user's profile Send private message
quark
Site Admin


Joined: 02 Nov 2007
Posts: 496

PostPosted: Thu Mar 13, 2014 4:38 pm    Post subject: Re: make: *** [Mult0.bo] Segmentation fault Reply with quote

Ted,

This output:
Code:
bsc -keep-fires -cross-info -sim -g mkMult0 -u Mult0.bsv
make: *** [Mult0.bo] Segmentation fault
would be more helpful if you run BSC with the -v flag, which instructs BSC to dump additional information as it executes. That would help identify whether the problem is starting BSC or occurs during execution of BSC (and at what particular stage during execution).

Also: The command "bsc" that you are running is probably a shell script that calls the actual "bsc" binary (or even a script that calls a script that calls the binary). On some systems, if the binary has execution problems, instead of seeing the output for that error, the script would instead segfault. If you're seeing a segfault, most likely it's not that BSC has segfaulted but that a system error has occurred and the script is masking it as a segfault. So I recommend locating the core binary and calling that directly. Probably like this:
Code:
$(BLUESPECDIR)/bin/linux64/bsc -v -sim -g mkMult0 -u Mult0.bsv

If you do this, do you see a better error message instead of the segfault?

Although you might want to confirm that the "bsc" script is calling the "linux64" version and not, say, the "linux32" version. You can check that with the "bsenv" script:
Code:
> bsenv platform
linux64

Hopefully, using -v and calling the binary directly will reveal the source of the error. For instance, one failure that can happen is that the core BSC binary might require certain versions of libraries. If you didn't have the right 64-bit libraries or didn't have the right GMP library version installed, then you might get an error like this:
Code:
bsc: error while loading shared libraries: libgmp.so.3: cannot open shared object file: No such file or directory
but the wrapper script could be masking this as a segfault.
Back to top
View user's profile Send private message
ted



Joined: 28 May 2013
Posts: 3

PostPosted: Fri Mar 14, 2014 2:38 pm    Post subject: make: *** [Mult0.bo] Segmentation fault Reply with quote

Thank you very much Quark, your suggestions helped me.
I knew that bsc was a wrapper script, but i didn't know exactly where
the binary that the script called was located. Following your suggestions,
I narrowed down the problem to a missing library file:

[email protected] mult0 2:31pm >make run
bsc -keep-fires -cross-info -v -sim -g mkMult0 -u Mult0.bsv
Bluespec Compiler, version 2013.05.beta2 (build 31258, 2013-05-21)
Copyright 2000-2013 Bluespec, Inc.
Parts copyright 2002, The University Court of the University of Glasgow.
Parts copyright 1982-1999 Lennart Augustsson, Thomas Johnsson,
Chalmers University of Technology.
Parts copyright 1999-2000, Daan Leijen.
Parts copyright 1991, 1999 Free Software Foundation, Inc.
Parts copyright 1995-2013, Regents of the University of Colorado.
Parts copyright 2010, Don Stewart.
All rights reserved.
See documentation for license details.

Invoking command line:
bsc -keep-fires -cross-info -v -sim -g mkMult0 -u Mult0.bsv

starting license
make: *** [Mult0.bo] Segmentation fault
[email protected] mult0 2:31pm >/CMC/tools/bluespec/lib/bin/linux64/bsc -keep-fires -cross-info -v -sim -g mkMult0 -u Mult0.bsv
/CMC/tools/bluespec/lib/bin/linux64/bsc: error while loading shared libraries: libyices.so.2: cannot open shared object file: No such file or directory


I'll get together with our system admins to follow up on this missing
libyices.so.2 file. I don't think that its expected to be in the bluespec
insgtalltion tree correct but rather in our /usr/lib directory correct?

Thank you very much for replying with a very clear and concise posting..
It is refreshing to see as one does not usually see this on forums...

Ted
Back to top
View user's profile Send private message
ted



Joined: 28 May 2013
Posts: 3

PostPosted: Tue Mar 18, 2014 12:44 pm    Post subject: bsc still reports segmentation fault Reply with quote

Hello,

We found the libyices.so.2 within the bluespec install tree and I modified
the bluespec.env so that LD_LIBRARY_PATH points to where it is found:

# T. Obuchowicz
# March 17, 2014
# set LD_LIBRARY_PATH to where the libyices.so.2 file
# is located within the bluespec install (needed under
# SL6)

if ($?LD_LIBRARY_PATH == 0) then
setenv LD_LIBRARY_PATH /CMC/tools/bluespec/lib/SAT/g++4_64:/CMC/tools/bluespec/
lib/SAT/g++4
else
setenv LD_LIBRARY_PATH /CMC/tools/bluespec/lib/SAT/g++4_64:/CMC/tools/bluespec/
lib/SAT/g++4:${LD_LIBRARY_PATH}
endif
When I run the bsc executable, it still seg faults:

[email protected] mult0 12:01pm >echo $LD_LIBRARY_PATH
/CMC/tools/bluespec/lib/SAT/g++4_64:/CMC/tools/bluespec/lib/SAT/g++4
[email protected] mult0 12:01pm >cadpass
[email protected] mult0 12:02pm >/CMC/tools/bluespec/lib/bin/linux64/bsc -keep-fires -cross-info -v -sim -g mkMult0 -u M
ult0.bsv
Bluespec Compiler, version 2013.05.beta2 (build 31258, 2013-05-21)
Copyright 2000-2013 Bluespec, Inc.
Parts copyright 2002, The University Court of the University of Glasgow.
Parts copyright 1982-1999 Lennart Augustsson, Thomas Johnsson,
Chalmers University of Technology.
Parts copyright 1999-2000, Daan Leijen.
Parts copyright 1991, 1999 Free Software Foundation, Inc.
Parts copyright 1995-2013, Regents of the University of Colorado.
Parts copyright 2010, Don Stewart.
All rights reserved.
See documentation for license details.

Invoking command line:
bsc -keep-fires -cross-info -v -sim -g mkMult0 -u Mult0.bsv
starting license
Segmentation fault


When I run the makefile in the mult0 example (which calls the bsc
wrapper script) :

[email protected] mult0 12:05pm >make run
bsc -keep-fires -cross-info -v -sim -g mkMult0 -u Mult0.bsv
Bluespec Compiler, version 2013.05.beta2 (build 31258, 2013-05-21)
Copyright 2000-2013 Bluespec, Inc.
Parts copyright 2002, The University Court of the University of Glasgow.
Parts copyright 1982-1999 Lennart Augustsson, Thomas Johnsson,
Chalmers University of Technology.
Parts copyright 1999-2000, Daan Leijen.
Parts copyright 1991, 1999 Free Software Foundation, Inc.
Parts copyright 1995-2013, Regents of the University of Colorado.
Parts copyright 2010, Don Stewart.
All rights reserved.
See documentation for license details.

Invoking command line:
bsc -keep-fires -cross-info -v -sim -g mkMult0 -u Mult0.bsv
starting license
make: *** [Mult0.bo] Segmentation fault


If I unsetenv LD_LIBRARY_PATH, it complains about the missing .so file
once agan:

[email protected] mult0 12:05pm >unsetenv LD_LIBRARY_PATH
[email protected] mult0 12:06pm >/CMC/tools/bluespec/lib/bin/linux64/bsc -keep-fires -cross-info -v -sim -g mkMult0 -u M
ult0.bsv
/CMC/tools/bluespec/lib/bin/linux64/bsc: error while loading shared libraries: libyices.so.2: cannot open shared objec
t file: No such file or directory



ldd reports that all library files required by the bsc executable are found:


[email protected] BLUESPEC 12:10pm >file /CMC/tools/bluespec/lib/bin/linux64/bsc
/CMC/tools/bluespec/lib/bin/linux64/bsc: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses
shared libs), for GNU/Linux 2.6.0, stripped
[email protected] BLUESPEC 12:11pm >ldd /CMC/tools/bluespec/lib/bin/linux64/bsc
linux-vdso.so.1 => (0x00007fff50d71000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00000033f8a00000)
libyices.so.2 => /CMC/tools/bluespec/lib/SAT/g++4_64/libyices.so.2 (0x00007fdc6d792000)
libstp.so.1 => /CMC/tools/bluespec/lib/SAT/g++4_64/libstp.so.1 (0x00007fdc6d38a000)
librt.so.1 => /lib64/librt.so.1 (0x00000033f9200000)
libutil.so.1 => /lib64/libutil.so.1 (0x0000003407200000)
libdl.so.2 => /lib64/libdl.so.2 (0x00000033f8600000)
libgmp.so.3 => /usr/lib64/libgmp.so.3 (0x00000033ff200000)
libm.so.6 => /lib64/libm.so.6 (0x00000033f8e00000)
libc.so.6 => /lib64/libc.so.6 (0x00000033f8200000)
/lib64/ld-linux-x86-64.so.2 (0x00000033f7e00000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000033fee00000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000033fea00000)


All the shared library files seem to be found and yet the bsc binary still
reports a "Segmentation Fault"

Does anyone know if version v2013.05.beta2 is certified for use
on Scientific Linux 6? Bluespec 2012a was running fine on our Scientifici
Linux 5 systems..

Thanks.

Ted
Back to top
View user's profile Send private message
quark
Site Admin


Joined: 02 Nov 2007
Posts: 496

PostPosted: Tue Jul 01, 2014 1:12 pm    Post subject: Re: bsc still reports segmentation fault Reply with quote

From this output, it appears that the segfault occurs when trying to check out the license. (The libyices library is not relevant to the segfault. Setting LD_LIBRARY_PATH is required if you're not using the wrapper, which sets the variable for you.)

The code for checking out the license is provided by a third party. It's possible that the version we use does not support Scientific Linux 6. One thing to check is whether you have a network device called "eth0". Newer Linux systems use a different naming convention. The Flex license software expects "eth0", so if yours has a different name, then you will need to change it to "eth0".

If that doesn't work, you can also try using a new release of BSC. In the Release forum, there is a new version 2014.06.A. I don't have any reason to believe that this will fix the problem, but it's worth a try (and worth upgrading anyway).
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    bluespec.com Forum Index -> Tools: BSC (Bluespec Compiler) All times are GMT - 4 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum
bluespec.com topic RSS feed 


Powered by phpBB © 2001, 2005 phpBB Group
Protected by Anti-Spam ACP